2024-03-25T17:51:07.665 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T17:51:07.665 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315/orig.config.yaml 2024-03-25T17:51:07.706 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:51:07.790 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621315' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFu75LBK5+mmCXqXFQkEaeqx9olToVK74M8E1goFv+hIIYcWKlNBJAmGrZuaHX/fnZlzu8o4Ef/fcApmHnY2UWA= smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPoGCLeTMdvJXt4cXBSaRu+M6PrJ8QIjxvJD4/CWoPQiXProONOev5+Tv0UkCP3AYZH/yTtc4C7WCDgX2lCfAyY= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T17:51:07.791 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T17:51:07.791 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T17:51:07.791 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T17:51:07.793 INFO:teuthology.task.internal:Checking packages... 2024-03-25T17:51:07.816 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T17:51:07.816 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T17:51:07.816 INFO:teuthology.packaging:ref: None 2024-03-25T17:51:07.816 INFO:teuthology.packaging:tag: None 2024-03-25T17:51:07.816 INFO:teuthology.packaging:branch: main 2024-03-25T17:51:07.816 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:51:07.816 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T17:51:08.157 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T17:51:08.159 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T17:51:08.165 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T17:51:08.166 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T17:51:08.175 INFO:teuthology.task.internal:Saving configuration 2024-03-25T17:51:08.190 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T17:51:08.197 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:51:08.225 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:13.771034', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFu75LBK5+mmCXqXFQkEaeqx9olToVK74M8E1goFv+hIIYcWKlNBJAmGrZuaHX/fnZlzu8o4Ef/fcApmHnY2UWA='} 2024-03-25T17:51:08.247 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:13.772582', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPoGCLeTMdvJXt4cXBSaRu+M6PrJ8QIjxvJD4/CWoPQiXProONOev5+Tv0UkCP3AYZH/yTtc4C7WCDgX2lCfAyY='} 2024-03-25T17:51:08.247 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T17:51:08.258 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T17:51:08.259 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T17:51:08.259 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T17:51:08.337 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ffbf69470d0>, signals=[15]) 2024-03-25T17:51:08.338 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T17:51:08.346 INFO:teuthology.task.internal:Opening connections... 2024-03-25T17:51:08.346 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-03-25T17:51:08.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:51:08.454 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2024-03-25T17:51:08.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:51:08.543 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T17:51:08.551 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-03-25T17:51:08.569 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-03-25T17:51:08.569 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:51:08.627 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:51:08.628 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:51:08.628 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:51:08.628 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-03-25T17:51:08.648 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2024-03-25T17:51:08.665 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2024-03-25T17:51:08.665 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2024-03-25T17:51:08.720 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="CentOS Stream" 2024-03-25T17:51:08.720 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="9" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:ID="centos" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="9" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:51:08.721 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:51:08.722 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2024-03-25T17:51:08.748 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T17:51:08.759 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T17:51:08.768 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T17:51:08.768 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:51:08.771 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:51:08.786 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T17:51:08.798 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T17:51:08.798 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:51:08.829 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:51:08.854 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T17:51:08.911 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T17:51:08.926 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T17:51:08.927 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T17:51:08.927 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:51:08.927 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:51:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:08.943 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-03-25T17:51:08.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:08.944 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-03-25T17:51:08.998 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:51:08.999 INFO:teuthology.task.kernel:Running kernel on smithi179: 5.14.0-361.el9.x86_64 2024-03-25T17:51:08.999 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum install -y kernel 2024-03-25T17:51:09.002 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:51:09.002 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-361.el9.x86_64 2024-03-25T17:51:09.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2024-03-25T17:51:11.101 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T17:51:11.123 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T17:51:12.501 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-25T17:51:12.574 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-25T17:51:18.963 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 1.3 MB/s | 8.0 MB 00:06 2024-03-25T17:51:20.526 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.0 MB 00:07 2024-03-25T17:51:28.368 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-25T17:51:29.646 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-25T17:51:40.944 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 811 kB/s | 6.1 MB 00:07 2024-03-25T17:51:41.445 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 663 kB/s | 6.1 MB 00:09 2024-03-25T17:51:42.458 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-25T17:51:43.125 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-25T17:51:44.242 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T17:51:44.655 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-25T17:51:49.075 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-25T17:51:49.836 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-25T17:51:51.249 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:51:51.250 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:51:51.340 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T17:51:51.345 INFO:teuthology.orchestra.run.smithi179.stdout:Install 4 Packages 2024-03-25T17:51:51.346 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T17:51:51.346 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 76 M 2024-03-25T17:51:51.346 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 123 M 2024-03-25T17:51:51.346 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T17:51:51.616 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:51.616 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:51.628 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:51.729 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:51.731 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:51.748 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:51.919 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:51.922 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:51.942 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:51.945 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:51.946 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:51.973 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:52.044 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:52.048 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:52.083 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:52.092 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:51:52.092 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:51:52.198 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T17:51:52.206 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T17:51:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-03-25T17:51:52.206 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T17:51:52.206 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T17:51:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:51:52.206 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-25T17:51:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:51:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:51:52.206 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:51:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T17:51:52.207 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T17:51:52.207 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T17:51:52.207 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-03-25T17:51:52.207 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T17:51:52.207 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 76 M 2024-03-25T17:51:52.207 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 123 M 2024-03-25T17:51:52.208 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T17:51:52.247 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:52.285 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:52.300 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:52.386 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:52.427 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:52.441 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:52.535 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:52.565 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:52.595 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:52.598 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:52.621 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:52.629 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:52.646 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:52.663 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:52.669 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:52.672 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.698 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.715 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.716 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.717 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.733 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.734 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.734 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.776 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.827 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.847 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.867 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:52.887 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:52.938 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:52.958 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:52.959 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:52.982 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:53.056 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:53.246 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:53.260 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:53.316 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:53.542 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:53.574 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:53.623 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:53.633 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:53.655 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:53.667 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:53.690 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:53.699 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:53.717 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:53.763 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:53.795 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:53.848 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:53.941 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:53.972 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:53.989 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:54.023 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:54.045 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:54.082 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:54.085 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:54.099 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:54.130 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:54.139 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:54.172 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:54.193 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:54.295 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:54.316 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:54.407 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:54.445 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:54.491 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:54.498 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:54.511 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:54.630 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:54.645 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:54.759 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:54.799 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:54.852 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:54.858 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:54.900 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:54.903 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:54.904 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:54.921 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:54.922 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:54.936 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:54.938 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:54.938 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:54.940 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:54.957 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:54.957 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:54.975 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:54.988 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:51:54.998 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:55.005 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:55.008 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:55.025 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.048 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:55.058 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:55.082 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:55.094 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.095 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:55.138 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:55.145 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:55.149 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.181 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.191 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.215 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:55.219 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:55.233 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.233 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.245 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:55.261 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:55.321 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.322 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.354 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.373 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.391 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:55.455 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:55.479 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:55.485 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:55.660 INFO:teuthology.orchestra.run.smithi179.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 16 kB/s | 66 kB 00:04 2024-03-25T17:51:55.747 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:55.789 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:55.991 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:56.006 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:56.028 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:56.100 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:56.119 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:56.217 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:56.396 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:56.410 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:56.416 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:56.492 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:56.560 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:56.694 INFO:teuthology.orchestra.run.smithi179.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 2.9 MB/s | 15 MB 00:05 2024-03-25T17:51:56.696 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:51:56.724 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:56.823 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:56.889 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:56.903 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:51:56.988 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:57.049 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:57.071 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:57.145 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:57.204 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:57.210 INFO:teuthology.orchestra.run.smithi179.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 5.9 MB/s | 33 MB 00:05 2024-03-25T17:51:57.265 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:57.480 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:57.514 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:57.554 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:57.597 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:57.781 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:58.112 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 5.7 MB/s | 33 MB 00:05 2024-03-25T17:51:58.113 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:58.202 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:51:58.226 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:58.256 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:58.261 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:51:58.391 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:58.542 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:51:58.545 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 2.4 MB/s | 15 MB 00:06 2024-03-25T17:51:58.570 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): kernel-5.14.0-432.el9.x86_64.rpm 10 kB/s | 66 kB 00:06 2024-03-25T17:51:58.571 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:51:58.588 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:58.644 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:58.703 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:58.703 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:58.721 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:58.802 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:51:58.809 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:58.822 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:58.848 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:51:59.217 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:51:59.230 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:59.234 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:59.395 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:51:59.450 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:51:59.845 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:51:59.976 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:00.041 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:00.110 INFO:teuthology.orchestra.run.smithi179.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 6.2 MB/s | 27 MB 00:04 2024-03-25T17:52:00.116 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:00.116 INFO:teuthology.orchestra.run.smithi179.stdout:Total 8.7 MB/s | 76 MB 00:08 2024-03-25T17:52:00.176 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:00.264 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:00.405 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:00.464 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:00.512 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:00.551 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:00.714 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:00.780 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T17:52:00.813 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:00.834 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:01.158 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T17:52:01.158 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T17:52:01.214 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:01.329 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:01.482 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:01.647 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:02.164 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T17:52:02.164 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T17:52:02.329 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 6.5 MB/s | 27 MB 00:04 2024-03-25T17:52:02.334 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:02.335 INFO:teuthology.orchestra.run.smithi052.stdout:Total 7.5 MB/s | 76 MB 00:10 2024-03-25T17:52:03.027 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T17:52:03.454 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T17:52:03.454 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T17:52:04.500 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T17:52:04.500 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T17:52:04.635 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T17:52:05.342 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:05.433 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:07.106 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T17:52:07.745 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:07.837 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:08.153 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:08.290 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:10.719 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:10.877 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:12.970 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:13.074 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:15.972 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:16.008 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:17.869 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:21.133 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:47.162 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:47.162 INFO:teuthology.orchestra.run.smithi179.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:52:47.162 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T17:52:47.207 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:47.594 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:47.595 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:47.595 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:47.595 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:48.014 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:48.014 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T17:52:48.014 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T17:52:48.014 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:48.014 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:48.014 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:52:48.015 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:48.015 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T17:52:48.015 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T17:52:48.359 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:52:48.471 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:48.471 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:52:48.471 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:52:48.471 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:52:48.471 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:52:48.472 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T17:52:48.472 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi179.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:52:48.472 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:52:48.551 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:48.551 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:52:48.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo rpm -qi grub2-tools 2024-03-25T17:52:48.585 INFO:teuthology.orchestra.run.smithi179.stdout:Name : grub2-tools 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Epoch : 1 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Version : 2.06 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Release : 68.el9 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: x86_64 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Group : Unspecified 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Size : 8264254 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:License : GPLv3+ 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:52:48.586 INFO:teuthology.orchestra.run.smithi179.stdout:Packager : builder@centos.org 2024-03-25T17:52:48.587 INFO:teuthology.orchestra.run.smithi179.stdout:Vendor : CentOS 2024-03-25T17:52:48.587 INFO:teuthology.orchestra.run.smithi179.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:52:48.587 INFO:teuthology.orchestra.run.smithi179.stdout:Summary : Support tools for GRUB. 2024-03-25T17:52:48.587 INFO:teuthology.orchestra.run.smithi179.stdout:Description : 2024-03-25T17:52:48.587 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T17:52:48.587 INFO:teuthology.orchestra.run.smithi179.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:52:48.587 INFO:teuthology.orchestra.run.smithi179.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:52:48.587 INFO:teuthology.orchestra.run.smithi179.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:52:48.587 INFO:teuthology.orchestra.run.smithi179.stdout:hardware devices. 2024-03-25T17:52:48.587 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T17:52:48.587 INFO:teuthology.orchestra.run.smithi179.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:52:48.588 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:52:48.588 INFO:teuthology.task.kernel:Updating grub on smithi179 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:52:48.588 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:52:49.220 INFO:teuthology.orchestra.run.smithi179.stderr:Generating grub configuration file ... 2024-03-25T17:52:50.611 INFO:teuthology.orchestra.run.smithi179.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:52:50.655 INFO:teuthology.orchestra.run.smithi179.stderr:done 2024-03-25T17:52:50.656 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-03-25T17:52:50.671 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.S9IqBGvNiC 2024-03-25T17:52:50.671 DEBUG:teuthology.orchestra.run.smithi179:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.S9IqBGvNiC 2024-03-25T17:52:50.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.S9IqBGvNiC 2024-03-25T17:52:50.883 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.S9IqBGvNiC is 6KB 2024-03-25T17:52:50.903 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.S9IqBGvNiC 2024-03-25T17:52:50.916 DEBUG:teuthology.orchestra.run.smithi179:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:52:50.981 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:52:50.981 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:52:50.981 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:52:50.981 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:52:50.983 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:52:51.128 DEBUG:teuthology.orchestra.run.smithi179:> sudo shutdown -r now 2024-03-25T17:52:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:51.894 INFO:teuthology.orchestra.run.smithi052.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:52:51.894 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T17:52:51.940 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:52.280 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:52.281 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:52.686 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T17:52:52.687 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T17:52:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:52:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:52:52.687 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T17:52:52.687 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T17:52:52.999 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:52:53.141 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:53.141 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:52:53.141 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:52:53.141 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:52:53.141 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:52:53.141 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T17:52:53.142 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:52:53.142 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:52:53.224 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:52:53.225 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:52:53.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 68.el9 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8264254 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:52:53.262 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T17:52:53.263 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:52:53.264 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:52:53.264 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:52:53.264 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:52:53.888 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2024-03-25T17:52:55.308 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:52:55.328 INFO:teuthology.orchestra.run.smithi052.stderr:done 2024-03-25T17:52:55.330 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-03-25T17:52:55.345 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.dEgz8XE4wD 2024-03-25T17:52:55.346 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dEgz8XE4wD 2024-03-25T17:52:55.419 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.dEgz8XE4wD 2024-03-25T17:52:55.569 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.dEgz8XE4wD is 6KB 2024-03-25T17:52:55.590 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.dEgz8XE4wD 2024-03-25T17:52:55.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:52:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:52:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:52:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:52:55.676 INFO:teuthology.orchestra.run.smithi052.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:52:55.677 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:52:55.832 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2024-03-25T17:53:21.149 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T17:53:21.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-25T17:53:21.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:53:25.838 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T17:53:25.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-25T17:53:25.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:21.199 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:54:25.842 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:54:30.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-25T17:54:30.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:30.656 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-03-25T17:54:31.063 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-25T17:54:31.063 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:54:31.063 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-03-25T17:54:31.077 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:54:31.077 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:54:31.077 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:54:31.077 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:54:32.078 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T17:54:32.078 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:54:32.145 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2024-03-25T17:54:32.182 DEBUG:teuthology.parallel:result is None 2024-03-25T17:54:34.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-25T17:54:34.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:35.217 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-03-25T17:54:35.640 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-25T17:54:35.640 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:54:35.640 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-03-25T17:54:35.654 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:54:35.654 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:54:35.655 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:54:35.655 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:54:36.655 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T17:54:36.655 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:54:36.716 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2024-03-25T17:54:36.756 DEBUG:teuthology.parallel:result is None 2024-03-25T17:54:36.756 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T17:54:36.763 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T17:54:36.763 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:54:36.765 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:54:36.790 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T17:54:36.800 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T17:54:36.811 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T17:54:36.812 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:54:36.817 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:54:36.879 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T17:54:36.887 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T17:54:36.888 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:54:36.890 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:54:36.919 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:54:36.923 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:54:36.932 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:54:36.936 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:54:36.938 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T17:54:36.946 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T17:54:36.946 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:54:36.973 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:54:37.002 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T17:54:37.011 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T17:54:37.011 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:54:37.037 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:54:37.055 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:54:37.122 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:54:37.167 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:54:37.244 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:54:37.283 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T17:54:37.283 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:54:37.347 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:54:37.389 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:54:37.460 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:54:37.502 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:54:37.543 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T17:54:37.543 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:54:37.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-03-25T17:54:37.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2024-03-25T17:54:37.677 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:54:37.762 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:54:38.000 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T17:54:38.011 INFO:teuthology.task.internal:Starting timer... 2024-03-25T17:54:38.011 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T17:54:38.027 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T17:54:38.045 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-03-25T17:54:38.108 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-03-25T17:54:38.110 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2024-03-25T17:54:38.180 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2024-03-25T17:54:38.182 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T17:54:38.182 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-03-25T17:54:38.204 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-03-25T17:54:38.204 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2024-03-25T17:54:38.225 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2024-03-25T17:54:38.225 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi052.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi179.front.sepia.ceph.com': 'permissive'} 2024-03-25T17:54:38.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:54:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:54:38.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:54:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:54:38.295 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T17:54:38.295 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:54:38.422 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:54:38.452 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T17:54:38.465 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T17:54:38.480 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T17:54:38.481 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T17:59:02.232 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2024-03-25T17:59:02.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-25T17:59:02.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:59:02.325 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-03-25T17:59:02.400 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-03-25T17:59:02.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-25T17:59:02.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:59:02.491 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-03-25T17:59:02.565 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-25T17:59:02.566 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T17:59:02.578 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T17:59:02.578 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:59:02.578 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:59:02.580 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:59:02.581 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:59:02.610 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:59:02.627 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:59:02.638 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:59:02.654 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:59:02.657 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-03-25T17:59:02.671 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-03-25T17:59:02.684 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: ntpd: command not found 2024-03-25T17:59:02.688 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:59:02.697 INFO:teuthology.orchestra.run.smithi179.stdout:506 Cannot talk to daemon 2024-03-25T17:59:02.704 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:59:02.714 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:59:02.730 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:59:02.752 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:59:02.755 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:59:02.755 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-03-25T17:59:02.756 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:02.756 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:02.756 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:02.756 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:02.784 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:59:02.788 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:59:02.788 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-03-25T17:59:02.788 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:02.788 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:02.788 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:02.788 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:02.789 INFO:teuthology.run_tasks:Running task install... 2024-03-25T17:59:02.797 DEBUG:teuthology.task.install:project ceph 2024-03-25T17:59:02.797 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T17:59:02.798 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T17:59:02.798 INFO:teuthology.task.install:Using flavor: default 2024-03-25T17:59:02.803 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T17:59:02.803 INFO:teuthology.task.install:extra packages: [] 2024-03-25T17:59:02.804 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:59:02.804 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:59:02.806 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:59:02.806 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:59:02.947 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:59:02.947 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:59:03.020 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:59:03.020 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:59:03.279 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:59:03.279 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T17:59:03.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:59:03.308 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:59:03.308 DEBUG:teuthology.orchestra.run.smithi179:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:59:03.343 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:59:03.343 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T17:59:03.343 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:59:03.373 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:59:03.373 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:59:03.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:59:03.443 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:59:03.455 DEBUG:teuthology.orchestra.run.smithi179:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:59:03.522 INFO:teuthology.orchestra.run.smithi179.stdout:check_obsoletes = 1 2024-03-25T17:59:03.523 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum clean all 2024-03-25T17:59:03.525 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:59:03.592 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2024-03-25T17:59:03.594 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2024-03-25T17:59:03.758 INFO:teuthology.orchestra.run.smithi179.stdout:74 files removed 2024-03-25T17:59:03.785 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-radosgw 2024-03-25T17:59:03.851 INFO:teuthology.orchestra.run.smithi052.stdout:74 files removed 2024-03-25T17:59:03.879 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw 2024-03-25T17:59:05.295 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 74 kB/s | 82 kB 00:01 2024-03-25T17:59:05.450 INFO:teuthology.orchestra.run.smithi179.stdout:ceph packages for x86_64 60 kB/s | 82 kB 00:01 2024-03-25T17:59:06.221 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 11 kB/s | 10 kB 00:00 2024-03-25T17:59:06.492 INFO:teuthology.orchestra.run.smithi179.stdout:ceph noarch packages 10 kB/s | 10 kB 00:00 2024-03-25T17:59:07.055 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T17:59:07.401 INFO:teuthology.orchestra.run.smithi179.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T17:59:07.682 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T17:59:08.094 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-25T17:59:09.474 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-25T17:59:09.803 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-25T17:59:14.003 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-25T17:59:18.902 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 939 kB/s | 8.0 MB 00:08 2024-03-25T17:59:21.320 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-25T17:59:24.400 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 2.0 MB/s | 19 MB 00:09 2024-03-25T17:59:32.715 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 800 kB/s | 6.1 MB 00:07 2024-03-25T17:59:34.521 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-25T17:59:35.422 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-25T17:59:36.695 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 727 kB/s | 6.1 MB 00:08 2024-03-25T17:59:38.200 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-25T17:59:39.704 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-25T17:59:40.271 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-03-25T17:59:42.785 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T17:59:42.803 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T17:59:42.803 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-25T17:59:42.803 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T17:59:42.803 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T17:59:42.803 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:59:42.803 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrading: 2024-03-25T17:59:42.803 INFO:teuthology.orchestra.run.smithi179.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:59:42.803 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:59:42.803 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-25T17:59:42.803 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options 2024-03-25T17:59:42.803 INFO:teuthology.orchestra.run.smithi179.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:59:42.804 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout:Installing weak dependencies: 2024-03-25T17:59:42.805 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T17:59:42.806 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T17:59:42.806 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T17:59:42.806 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T17:59:42.806 INFO:teuthology.orchestra.run.smithi179.stdout:Install 30 Packages 2024-03-25T17:59:42.806 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrade 2 Packages 2024-03-25T17:59:42.806 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T17:59:42.806 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 58 M 2024-03-25T17:59:42.806 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T17:59:44.223 INFO:teuthology.orchestra.run.smithi179.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.3 MB/s | 5.3 MB 00:01 2024-03-25T17:59:44.273 INFO:teuthology.orchestra.run.smithi179.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 505 kB/s | 25 kB 00:00 2024-03-25T17:59:44.406 INFO:teuthology.orchestra.run.smithi179.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.4 MB/s | 742 kB 00:00 2024-03-25T17:59:44.498 INFO:teuthology.orchestra.run.smithi179.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.3 MB/s | 498 kB 00:00 2024-03-25T17:59:44.802 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T17:59:45.140 INFO:teuthology.orchestra.run.smithi179.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 7.9 MB/s | 5.1 MB 00:00 2024-03-25T17:59:45.215 INFO:teuthology.orchestra.run.smithi179.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 602 kB/s | 45 kB 00:00 2024-03-25T17:59:45.307 INFO:teuthology.orchestra.run.smithi179.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.4 MB/s | 131 kB 00:00 2024-03-25T17:59:45.499 INFO:teuthology.orchestra.run.smithi179.stdout:(8/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 7.9 MB/s | 20 MB 00:02 2024-03-25T17:59:45.524 INFO:teuthology.orchestra.run.smithi179.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 746 kB/s | 162 kB 00:00 2024-03-25T17:59:45.558 INFO:teuthology.orchestra.run.smithi179.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 5.4 MB/s | 320 kB 00:00 2024-03-25T17:59:45.583 INFO:teuthology.orchestra.run.smithi179.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-25T17:59:45.617 INFO:teuthology.orchestra.run.smithi179.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-25T17:59:45.776 INFO:teuthology.orchestra.run.smithi179.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 551 kB/s | 106 kB 00:00 2024-03-25T17:59:45.801 INFO:teuthology.orchestra.run.smithi179.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-25T17:59:45.826 INFO:teuthology.orchestra.run.smithi179.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-25T17:59:45.894 INFO:teuthology.orchestra.run.smithi179.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-03-25T17:59:45.919 INFO:teuthology.orchestra.run.smithi179.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-25T17:59:45.944 INFO:teuthology.orchestra.run.smithi179.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 829 kB/s | 41 kB 00:00 2024-03-25T17:59:46.106 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:59:46.106 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:59:46.503 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:59:46.539 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:59:46.706 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:59:46.728 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:59:46.729 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:59:46.749 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:59:46.779 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:59:46.796 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:59:46.873 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:59:46.896 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:59:46.920 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:59:46.938 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:59:47.010 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:59:47.029 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:59:47.186 INFO:teuthology.orchestra.run.smithi179.stdout:(19/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 2.3 MB/s | 9.7 MB 00:04 2024-03-25T17:59:47.366 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:59:47.379 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:59:47.380 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:59:47.380 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:59:47.380 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:59:47.380 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:59:47.380 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:59:47.380 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:59:47.380 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:59:47.380 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:59:47.380 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:59:47.380 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:59:47.381 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:59:47.381 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:59:47.381 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:59:47.381 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse 2024-03-25T17:59:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:59:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:59:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:59:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:59:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:59:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:59:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:59:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T17:59:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T17:59:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:59:47.382 INFO:teuthology.orchestra.run.smithi052.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:59:47.383 INFO:teuthology.orchestra.run.smithi052.stdout:Installing weak dependencies: 2024-03-25T17:59:47.383 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T17:59:47.383 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T17:59:47.383 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T17:59:47.383 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T17:59:47.383 INFO:teuthology.orchestra.run.smithi052.stdout:Install 30 Packages 2024-03-25T17:59:47.383 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 2 Packages 2024-03-25T17:59:47.383 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T17:59:47.383 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 58 M 2024-03-25T17:59:47.383 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T17:59:47.416 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:59:47.653 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:59:47.683 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:59:47.782 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:59:47.899 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:59:47.916 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:59:47.962 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:59:48.021 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:59:48.023 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:59:48.038 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:48.040 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:48.096 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:59:48.105 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:48.114 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:48.170 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:59:48.179 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:59:48.292 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:59:48.315 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:59:48.328 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:59:48.350 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:59:48.393 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:59:48.417 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:59:48.475 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:59:48.524 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:59:48.554 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:59:48.560 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:59:48.683 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:59:48.714 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:59:49.032 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:59:49.035 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:59:49.095 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:59:49.195 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:59:49.235 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:59:49.275 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:59:49.325 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:59:49.327 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:49.360 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:59:49.365 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:49.375 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:49.377 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:49.413 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:59:49.413 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:49.430 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:59:49.455 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:49.513 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:59:49.527 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:59:49.531 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:59:49.590 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:59:49.619 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:59:49.651 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:59:49.698 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:59:49.727 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:59:49.763 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:59:49.797 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:59:49.852 INFO:teuthology.orchestra.run.smithi179.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 9.2 kB/s | 36 kB 00:03 2024-03-25T17:59:49.928 INFO:teuthology.orchestra.run.smithi179.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 55 kB/s | 221 kB 00:04 2024-03-25T17:59:49.966 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:59:50.127 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:59:50.153 INFO:teuthology.orchestra.run.smithi179.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-25T17:59:50.211 INFO:teuthology.orchestra.run.smithi179.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-03-25T17:59:50.303 INFO:teuthology.orchestra.run.smithi179.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 710 kB/s | 49 kB 00:00 2024-03-25T17:59:50.370 INFO:teuthology.orchestra.run.smithi179.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-25T17:59:50.437 INFO:teuthology.orchestra.run.smithi179.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-03-25T17:59:50.496 INFO:teuthology.orchestra.run.smithi179.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-25T17:59:50.529 INFO:teuthology.orchestra.run.smithi179.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.1 MB/s | 838 kB 00:00 2024-03-25T17:59:50.531 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:59:50.580 INFO:teuthology.orchestra.run.smithi179.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-25T17:59:50.681 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:59:50.756 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:59:50.808 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:50.825 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:59:50.859 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:59:50.936 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:59:51.025 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:59:51.063 INFO:teuthology.orchestra.run.smithi179.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:59:51.105 INFO:teuthology.orchestra.run.smithi179.stdout:(30/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 12 kB/s | 47 kB 00:03 2024-03-25T17:59:57.486 INFO:teuthology.orchestra.run.smithi052.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 571 kB/s | 5.3 MB 00:09 2024-03-25T17:59:57.936 INFO:teuthology.orchestra.run.smithi052.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 56 kB/s | 25 kB 00:00 2024-03-25T17:59:58.721 INFO:teuthology.orchestra.run.smithi179.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 395 kB/s | 3.1 MB 00:08 2024-03-25T18:00:00.761 INFO:teuthology.orchestra.run.smithi052.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 263 kB/s | 742 kB 00:02 2024-03-25T18:00:02.245 INFO:teuthology.orchestra.run.smithi052.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 336 kB/s | 498 kB 00:01 2024-03-25T18:00:02.679 INFO:teuthology.orchestra.run.smithi179.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 284 kB/s | 3.4 MB 00:12 2024-03-25T18:00:02.687 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:02.687 INFO:teuthology.orchestra.run.smithi179.stdout:Total 2.9 MB/s | 58 MB 00:19 2024-03-25T18:00:02.720 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:00:02.758 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:00:02.759 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:00:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:00:03.225 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:00:04.687 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:00:04.760 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:00:04.801 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:00:04.826 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:00:04.849 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:00:04.866 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:00:05.058 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:00:05.112 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:00:05.430 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:00:05.503 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:00:05.578 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:00:05.664 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:00:05.688 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:00:05.750 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:00:05.779 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:00:05.822 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:00:05.890 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:00:05.915 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:00:05.979 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:00:06.014 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:00:06.121 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:00:06.156 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:00:06.197 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:00:06.229 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:00:06.258 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:00:06.291 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:00:06.619 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:00:06.796 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:00:06.868 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:00:06.900 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:00:06.943 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:00:07.401 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:00:07.432 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:00:07.503 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:00:07.570 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:00:07.601 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:00:09.751 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:00:09.789 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:00:10.343 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:00:10.567 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:00:10.641 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:00:10.641 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:00:10.641 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:00:10.641 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:00:10.677 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:00:18.094 INFO:teuthology.orchestra.run.smithi052.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 330 kB/s | 5.1 MB 00:15 2024-03-25T18:00:18.344 INFO:teuthology.orchestra.run.smithi052.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 180 kB/s | 45 kB 00:00 2024-03-25T18:00:18.827 INFO:teuthology.orchestra.run.smithi052.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 272 kB/s | 131 kB 00:00 2024-03-25T18:00:19.477 INFO:teuthology.orchestra.run.smithi052.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 249 kB/s | 162 kB 00:00 2024-03-25T18:00:20.069 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:00:20.069 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /sys 2024-03-25T18:00:20.069 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /proc 2024-03-25T18:00:20.070 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /mnt 2024-03-25T18:00:20.070 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /var/tmp 2024-03-25T18:00:20.070 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /home 2024-03-25T18:00:20.070 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /root 2024-03-25T18:00:20.070 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /tmp 2024-03-25T18:00:20.070 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:00:20.936 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:00:20.960 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:00:20.960 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:00:20.960 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:00:20.960 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:00:20.960 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:00:20.960 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:00:20.962 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:00:21.055 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:00:21.055 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:00:21.161 INFO:teuthology.orchestra.run.smithi052.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 190 kB/s | 320 kB 00:01 2024-03-25T18:00:22.187 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:00:22.187 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:00:22.188 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:00:22.189 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:00:22.189 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:00:22.189 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:00:22.189 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:00:22.189 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:00:22.189 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:00:22.189 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:00:22.189 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:00:22.189 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:00:22.189 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:00:22.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:00:22.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:00:22.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:00:22.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:00:22.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:00:22.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:00:22.190 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:00:22.461 INFO:teuthology.orchestra.run.smithi052.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 229 kB/s | 298 kB 00:01 2024-03-25T18:00:22.995 INFO:teuthology.orchestra.run.smithi052.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 186 kB/s | 99 kB 00:00 2024-03-25T18:00:23.103 INFO:teuthology.orchestra.run.smithi052.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T18:00:23.154 INFO:teuthology.orchestra.run.smithi052.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:00:23.179 INFO:teuthology.orchestra.run.smithi052.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-25T18:00:23.237 INFO:teuthology.orchestra.run.smithi052.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:00:23.279 INFO:teuthology.orchestra.run.smithi052.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:00:23.305 INFO:teuthology.orchestra.run.smithi052.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-25T18:00:23.323 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:00:23.323 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:00:23.323 INFO:teuthology.orchestra.run.smithi179.stdout:Upgraded: 2024-03-25T18:00:23.323 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.323 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.323 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:00:23.324 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:00:23.325 INFO:teuthology.orchestra.run.smithi179.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:00:23.326 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:00:23.326 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:00:23.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-test 2024-03-25T18:00:23.834 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:24.133 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:44 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:00:24.199 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:24.263 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:24.283 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:00:24.284 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:00:24.284 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-25T18:00:24.284 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout:Install 6 Packages 2024-03-25T18:00:24.285 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:00:24.286 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 48 M 2024-03-25T18:00:24.286 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 204 M 2024-03-25T18:00:24.286 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:00:24.432 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:24.488 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:24.574 INFO:teuthology.orchestra.run.smithi179.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-25T18:00:24.588 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:24.633 INFO:teuthology.orchestra.run.smithi179.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-25T18:00:24.653 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:24.675 INFO:teuthology.orchestra.run.smithi179.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T18:00:24.812 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:24.965 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:24.999 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:25.041 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:25.117 INFO:teuthology.orchestra.run.smithi179.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 424 kB/s | 187 kB 00:00 2024-03-25T18:00:25.172 INFO:teuthology.orchestra.run.smithi052.stdout:(18/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 268 kB/s | 9.7 MB 00:37 2024-03-25T18:00:25.172 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:25.334 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:25.517 INFO:teuthology.orchestra.run.smithi179.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 143 kB/s | 163 kB 00:01 2024-03-25T18:00:25.656 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:25.698 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:25.717 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:25.790 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:25.824 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:00:25.965 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:26.005 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:26.040 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:26.072 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:26.075 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:00:26.162 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T18:00:26.213 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:26.239 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:00:26.259 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:26.377 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:26.379 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T18:00:26.443 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:26.545 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:26.568 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:26.690 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:26.708 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:26.748 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:26.837 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:26.919 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:26.997 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:27.005 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:27.026 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:27.163 INFO:teuthology.orchestra.run.smithi052.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 57 kB/s | 221 kB 00:03 2024-03-25T18:00:27.349 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:27.408 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:27.476 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:27.512 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:00:27.653 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:27.654 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:27.752 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:27.830 INFO:teuthology.orchestra.run.smithi052.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 13 kB/s | 36 kB 00:02 2024-03-25T18:00:27.982 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:00:28.052 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:28.184 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:28.230 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:28.305 INFO:teuthology.orchestra.run.smithi052.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 651 kB/s | 309 kB 00:00 2024-03-25T18:00:28.329 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:00:28.394 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:28.523 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:00:28.659 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:28.680 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:00:28.756 INFO:teuthology.orchestra.run.smithi052.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-03-25T18:00:28.776 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:28.848 INFO:teuthology.orchestra.run.smithi052.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-25T18:00:28.869 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:28.940 INFO:teuthology.orchestra.run.smithi052.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 537 kB/s | 49 kB 00:00 2024-03-25T18:00:29.048 INFO:teuthology.orchestra.run.smithi052.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-03-25T18:00:29.049 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:29.249 INFO:teuthology.orchestra.run.smithi052.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-03-25T18:00:29.382 INFO:teuthology.orchestra.run.smithi052.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-25T18:00:29.442 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:00:29.490 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:29.568 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:00:29.602 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:00:29.799 INFO:teuthology.orchestra.run.smithi052.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 3.8 MB/s | 1.6 MB 00:00 2024-03-25T18:00:29.801 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:29.896 INFO:teuthology.orchestra.run.smithi052.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:00:30.058 INFO:teuthology.orchestra.run.smithi052.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 16 kB/s | 47 kB 00:02 2024-03-25T18:00:34.966 INFO:teuthology.orchestra.run.smithi052.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 669 kB/s | 3.4 MB 00:05 2024-03-25T18:00:42.307 INFO:teuthology.orchestra.run.smithi052.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 262 kB/s | 3.1 MB 00:12 2024-03-25T18:00:58.564 INFO:teuthology.orchestra.run.smithi052.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 288 kB/s | 20 MB 01:10 2024-03-25T18:00:58.569 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:58.569 INFO:teuthology.orchestra.run.smithi052.stdout:Total 832 kB/s | 58 MB 01:11 2024-03-25T18:00:58.600 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:00:58.641 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:00:58.641 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:00:59.134 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:00:59.134 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:01:00.635 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:01:00.721 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:01:00.751 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:01:00.777 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:01:00.800 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:01:00.816 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:01:00.959 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:01:01.023 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:01:01.407 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:01:01.438 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:01.530 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:01.620 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:01:01.647 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:01:01.702 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:01:01.731 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:01:01.775 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:01:01.836 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:01:01.858 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:01:01.905 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:01:01.940 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:01:02.051 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:01:02.083 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:01:02.115 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:01:02.148 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:01:02.177 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:01:02.209 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:01:02.541 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:01:02.564 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:01:02.655 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:01:02.685 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:01:02.730 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:01:03.209 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:01:03.233 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:01:03.330 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:01:03.400 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:01:03.436 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:01:05.686 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:05.724 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:06.296 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:06.336 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:01:06.419 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:01:06.419 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:01:06.419 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:01:06.419 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:01:06.454 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:16.267 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:16.267 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2024-03-25T18:01:16.267 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2024-03-25T18:01:16.267 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2024-03-25T18:01:16.268 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2024-03-25T18:01:16.268 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2024-03-25T18:01:16.268 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /root 2024-03-25T18:01:16.268 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2024-03-25T18:01:16.268 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:01:17.204 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:01:17.228 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:01:17.228 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:01:17.228 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:01:17.228 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:01:17.228 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:01:17.228 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:01:17.230 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:01:17.315 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:01:17.315 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:18.416 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:18.416 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:01:18.416 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:01:18.416 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:01:18.416 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:01:18.416 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:01:18.416 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:01:18.417 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:01:18.418 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:01:19.578 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:19.578 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:01:19.578 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2024-03-25T18:01:19.578 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.578 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:01:19.579 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:01:19.580 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:01:19.581 INFO:teuthology.orchestra.run.smithi052.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:01:19.581 INFO:teuthology.orchestra.run.smithi052.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:01:19.581 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:01:19.581 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:01:19.838 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test 2024-03-25T18:01:20.403 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:36 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:01:20.563 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:01:20.564 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:01:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-25T18:01:20.564 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:01:20.564 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:01:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:01:20.564 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-25T18:01:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:01:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:01:20.564 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:01:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:01:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:01:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:01:20.565 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:01:20.565 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:01:20.565 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-03-25T18:01:20.565 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:01:20.565 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 48 M 2024-03-25T18:01:20.566 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 204 M 2024-03-25T18:01:20.566 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:01:20.821 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T18:01:20.896 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.7 MB/s | 300 kB 00:00 2024-03-25T18:01:20.930 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T18:01:21.022 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-25T18:01:21.705 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 159 kB/s | 163 kB 00:01 2024-03-25T18:03:06.863 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 456 kB/s | 47 MB 01:46 2024-03-25T18:03:06.864 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:06.865 INFO:teuthology.orchestra.run.smithi052.stdout:Total 464 kB/s | 48 MB 01:46 2024-03-25T18:03:06.877 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:03:06.898 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:03:06.898 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:03:07.288 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:03:07.288 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:03:07.894 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:03:07.954 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:03:07.997 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:03:08.037 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:03:08.086 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:03:08.106 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:03:12.842 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:03:12.916 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:03:15.323 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:03:15.323 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:03:15.323 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:03:15.323 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:03:15.323 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:03:15.323 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:03:15.744 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:03:15.744 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:03:15.744 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:03:15.744 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:15.745 INFO:teuthology.orchestra.run.smithi052.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:03:15.745 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:15.745 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:03:15.745 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:03:15.745 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:03:15.745 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:03:15.745 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:03:16.038 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph 2024-03-25T18:03:16.609 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:03:32 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:03:16.767 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:03:16.770 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:03:16.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:03:16.772 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:03:16.773 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:03:16.773 INFO:teuthology.orchestra.run.smithi052.stdout:========================================================================================= 2024-03-25T18:03:16.773 INFO:teuthology.orchestra.run.smithi052.stdout:Install 36 Packages 2024-03-25T18:03:16.773 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:03:16.773 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 30 M 2024-03-25T18:03:16.773 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 107 M 2024-03-25T18:03:16.773 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:03:17.588 INFO:teuthology.orchestra.run.smithi052.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T18:03:22.905 INFO:teuthology.orchestra.run.smithi052.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 263 kB/s | 1.5 MB 00:05 2024-03-25T18:03:23.413 INFO:teuthology.orchestra.run.smithi052.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 345 kB/s | 2.1 MB 00:06 2024-03-25T18:03:24.530 INFO:teuthology.orchestra.run.smithi052.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 222 kB/s | 247 kB 00:01 2024-03-25T18:03:24.647 INFO:teuthology.orchestra.run.smithi052.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:03:24.689 INFO:teuthology.orchestra.run.smithi052.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:03:24.714 INFO:teuthology.orchestra.run.smithi052.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:03:24.748 INFO:teuthology.orchestra.run.smithi052.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T18:03:24.798 INFO:teuthology.orchestra.run.smithi052.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-25T18:03:24.840 INFO:teuthology.orchestra.run.smithi052.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T18:03:24.915 INFO:teuthology.orchestra.run.smithi052.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:03:24.957 INFO:teuthology.orchestra.run.smithi052.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T18:03:24.990 INFO:teuthology.orchestra.run.smithi052.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T18:03:25.033 INFO:teuthology.orchestra.run.smithi052.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:03:25.233 INFO:teuthology.orchestra.run.smithi052.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 625 kB/s | 125 kB 00:00 2024-03-25T18:03:25.491 INFO:teuthology.orchestra.run.smithi052.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-25T18:03:25.900 INFO:teuthology.orchestra.run.smithi052.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 370 kB/s | 151 kB 00:00 2024-03-25T18:03:25.992 INFO:teuthology.orchestra.run.smithi052.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 474 kB/s | 43 kB 00:00 2024-03-25T18:03:26.167 INFO:teuthology.orchestra.run.smithi052.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 983 kB/s | 172 kB 00:00 2024-03-25T18:03:26.334 INFO:teuthology.orchestra.run.smithi052.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-25T18:03:26.426 INFO:teuthology.orchestra.run.smithi052.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-25T18:03:26.527 INFO:teuthology.orchestra.run.smithi052.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-03-25T18:03:26.618 INFO:teuthology.orchestra.run.smithi052.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 253 kB/s | 23 kB 00:00 2024-03-25T18:03:26.752 INFO:teuthology.orchestra.run.smithi052.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 146 kB/s | 19 kB 00:00 2024-03-25T18:03:26.869 INFO:teuthology.orchestra.run.smithi052.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2024-03-25T18:03:26.986 INFO:teuthology.orchestra.run.smithi052.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 398 kB/s | 46 kB 00:00 2024-03-25T18:03:27.086 INFO:teuthology.orchestra.run.smithi052.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 789 kB/s | 79 kB 00:00 2024-03-25T18:03:27.186 INFO:teuthology.orchestra.run.smithi052.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-25T18:03:27.278 INFO:teuthology.orchestra.run.smithi052.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-03-25T18:03:27.379 INFO:teuthology.orchestra.run.smithi052.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 899 kB/s | 90 kB 00:00 2024-03-25T18:03:27.479 INFO:teuthology.orchestra.run.smithi052.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 358 kB/s | 36 kB 00:00 2024-03-25T18:03:27.587 INFO:teuthology.orchestra.run.smithi052.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-25T18:03:27.696 INFO:teuthology.orchestra.run.smithi052.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.9 MB/s | 427 kB 00:00 2024-03-25T18:03:27.788 INFO:teuthology.orchestra.run.smithi052.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-25T18:03:35.854 INFO:teuthology.orchestra.run.smithi052.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 263 kB/s | 4.7 MB 00:18 2024-03-25T18:03:44.097 INFO:teuthology.orchestra.run.smithi179.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 242 kB/s | 47 MB 03:19 2024-03-25T18:03:44.099 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:44.099 INFO:teuthology.orchestra.run.smithi179.stdout:Total 247 kB/s | 48 MB 03:19 2024-03-25T18:03:44.111 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:03:44.131 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:03:44.131 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:03:44.513 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:03:44.513 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:03:45.098 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:03:45.179 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:03:45.230 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:03:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:03:45.322 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:03:45.349 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:03:49.890 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:03:52.323 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:03:52.323 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:03:52.323 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:03:52.323 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:03:52.323 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:03:52.323 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:03:52.657 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:03:52.657 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:03:52.657 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:03:52.657 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:52.657 INFO:teuthology.orchestra.run.smithi179.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:03:52.657 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:52.658 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:03:52.658 INFO:teuthology.orchestra.run.smithi179.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:03:52.658 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:03:52.658 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:03:52.658 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:03:52.995 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph 2024-03-25T18:03:53.562 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:04:13 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:03:53.716 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:03:53.718 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:03:53.719 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:03:53.720 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:03:53.721 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:03:53.722 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:03:53.722 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-25T18:03:53.722 INFO:teuthology.orchestra.run.smithi179.stdout:Install 36 Packages 2024-03-25T18:03:53.722 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:03:53.722 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 30 M 2024-03-25T18:03:53.722 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 107 M 2024-03-25T18:03:53.722 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:03:54.924 INFO:teuthology.orchestra.run.smithi179.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 13 kB/s | 6.6 kB 00:00 2024-03-25T18:03:58.191 INFO:teuthology.orchestra.run.smithi179.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 578 kB/s | 2.1 MB 00:03 2024-03-25T18:04:02.441 INFO:teuthology.orchestra.run.smithi179.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 190 kB/s | 1.5 MB 00:08 2024-03-25T18:04:03.341 INFO:teuthology.orchestra.run.smithi179.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 275 kB/s | 247 kB 00:00 2024-03-25T18:04:03.458 INFO:teuthology.orchestra.run.smithi179.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:04:03.491 INFO:teuthology.orchestra.run.smithi179.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.3 MB/s | 144 kB 00:00 2024-03-25T18:04:03.517 INFO:teuthology.orchestra.run.smithi179.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:04:03.542 INFO:teuthology.orchestra.run.smithi179.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-25T18:04:03.575 INFO:teuthology.orchestra.run.smithi179.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T18:04:03.617 INFO:teuthology.orchestra.run.smithi179.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T18:04:03.693 INFO:teuthology.orchestra.run.smithi179.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:04:03.726 INFO:teuthology.orchestra.run.smithi179.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:04:03.776 INFO:teuthology.orchestra.run.smithi179.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T18:04:03.810 INFO:teuthology.orchestra.run.smithi179.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T18:04:03.910 INFO:teuthology.orchestra.run.smithi179.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.2 MB/s | 125 kB 00:00 2024-03-25T18:04:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-03-25T18:04:04.519 INFO:teuthology.orchestra.run.smithi179.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 386 kB/s | 151 kB 00:00 2024-03-25T18:04:04.620 INFO:teuthology.orchestra.run.smithi179.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 435 kB/s | 43 kB 00:00 2024-03-25T18:04:04.787 INFO:teuthology.orchestra.run.smithi179.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-03-25T18:04:04.953 INFO:teuthology.orchestra.run.smithi179.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-25T18:04:05.054 INFO:teuthology.orchestra.run.smithi179.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-25T18:04:05.146 INFO:teuthology.orchestra.run.smithi179.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-25T18:04:05.246 INFO:teuthology.orchestra.run.smithi179.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 232 kB/s | 23 kB 00:00 2024-03-25T18:04:05.346 INFO:teuthology.orchestra.run.smithi179.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 195 kB/s | 19 kB 00:00 2024-03-25T18:04:05.446 INFO:teuthology.orchestra.run.smithi179.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-25T18:04:05.547 INFO:teuthology.orchestra.run.smithi179.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-03-25T18:04:05.647 INFO:teuthology.orchestra.run.smithi179.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 789 kB/s | 79 kB 00:00 2024-03-25T18:04:05.747 INFO:teuthology.orchestra.run.smithi179.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-25T18:04:05.848 INFO:teuthology.orchestra.run.smithi179.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-25T18:04:05.948 INFO:teuthology.orchestra.run.smithi179.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 900 kB/s | 90 kB 00:00 2024-03-25T18:04:06.040 INFO:teuthology.orchestra.run.smithi179.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 390 kB/s | 36 kB 00:00 2024-03-25T18:04:06.149 INFO:teuthology.orchestra.run.smithi179.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-25T18:04:06.250 INFO:teuthology.orchestra.run.smithi179.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-03-25T18:04:06.333 INFO:teuthology.orchestra.run.smithi179.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 240 kB/s | 20 kB 00:00 2024-03-25T18:04:07.468 INFO:teuthology.orchestra.run.smithi052.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 386 kB/s | 17 MB 00:44 2024-03-25T18:04:07.472 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:07.472 INFO:teuthology.orchestra.run.smithi052.stdout:Total 599 kB/s | 30 MB 00:50 2024-03-25T18:04:07.496 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:04:07.542 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:04:07.542 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:04:07.979 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:04:07.979 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:04:09.336 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:04:09.403 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:04:09.437 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:04:09.472 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:04:09.546 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:04:09.588 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:04:09.629 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:04:09.725 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:04:09.803 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:04:09.850 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:04:09.926 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:04:09.975 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:04:10.013 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:04:10.071 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:04:10.135 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:04:10.329 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:04:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:04:10.413 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:04:10.448 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:04:10.478 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:04:10.549 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:04:10.580 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:04:10.613 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:04:10.654 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:04:10.697 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:04:10.733 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:04:10.824 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:04:10.862 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:04:10.937 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:04:11.087 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:04:11.212 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:04:11.378 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:04:11.416 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:04:11.441 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:04:11.442 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:11.442 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:04:11.442 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:04:11.442 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:04:11.442 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:11.949 INFO:teuthology.orchestra.run.smithi179.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 282 kB/s | 4.7 MB 00:17 2024-03-25T18:04:12.974 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:04:13.001 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:04:13.001 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:13.001 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:04:13.001 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:04:13.001 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:04:13.002 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:13.467 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:04:13.489 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:04:13.489 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:13.489 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:04:13.490 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:04:13.490 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:04:13.490 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:13.708 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:04:13.731 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:04:13.731 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:13.731 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:04:13.731 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:04:13.731 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:04:13.732 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:13.800 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:04:15.436 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:04:15.437 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:04:15.437 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:04:15.437 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:04:15.437 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:04:15.437 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:04:15.437 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:04:15.437 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:04:15.437 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:04:15.437 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:04:15.437 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:04:15.437 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:04:15.438 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:04:15.438 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:04:15.438 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:04:15.438 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:04:15.438 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:04:15.438 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:04:15.438 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:04:15.438 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:04:15.438 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:04:15.438 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:04:15.438 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:04:15.439 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:04:15.439 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:04:15.439 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:04:15.440 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:04:15.440 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:04:15.440 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:04:15.440 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:04:15.440 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:04:15.441 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:04:15.441 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:04:15.441 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:04:15.441 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:04:15.441 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:04:16.513 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:04:16.513 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:16.513 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:04:16.513 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:16.513 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:16.513 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:16.514 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:16.514 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:16.514 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:16.514 INFO:teuthology.orchestra.run.smithi052.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:04:16.514 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:04:16.514 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:04:16.514 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:04:16.514 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:04:16.514 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:04:16.514 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:04:16.514 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:04:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:04:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:04:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:04:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:04:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:04:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:04:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:04:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:04:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:04:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:04:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:04:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:04:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:04:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:04:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:04:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:04:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:04:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:04:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:04:16.517 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:04:16.517 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:04:16.517 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:04:16.517 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:16.517 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:04:16.814 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base 2024-03-25T18:04:17.391 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:04:33 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:04:17.491 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:04:17.546 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:04:17.547 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-25T18:04:17.547 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:04:17.621 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install cephadm 2024-03-25T18:04:18.189 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:04:34 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:04:18.344 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:04:18.345 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:04:18.345 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-25T18:04:18.345 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:04:18.345 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:04:18.345 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:04:18.346 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:18.346 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:04:18.346 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:04:18.346 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:04:18.346 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:18.346 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 747 k 2024-03-25T18:04:18.346 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 754 k 2024-03-25T18:04:18.346 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:04:20.691 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 319 kB/s | 747 kB 00:02 2024-03-25T18:04:20.691 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:20.692 INFO:teuthology.orchestra.run.smithi052.stdout:Total 318 kB/s | 747 kB 00:02 2024-03-25T18:04:20.692 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:04:20.699 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:04:20.699 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:04:20.705 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:04:20.705 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:04:20.780 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:04:21.186 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:21.282 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:21.655 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:21.969 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:21.969 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:21.970 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:04:21.970 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:21.970 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:21.970 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:04:22.197 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:04:22.774 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:04:22.930 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:04:22.931 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:04:22.931 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T18:04:22.931 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:04:22.931 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:04:22.931 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:04:22.931 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:22.931 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:04:22.931 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:04:22.931 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:04:22.931 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:22.931 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 145 k 2024-03-25T18:04:22.932 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 436 k 2024-03-25T18:04:22.932 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:04:23.838 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 160 kB/s | 145 kB 00:00 2024-03-25T18:04:23.838 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:23.839 INFO:teuthology.orchestra.run.smithi052.stdout:Total 160 kB/s | 145 kB 00:00 2024-03-25T18:04:23.839 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:04:23.848 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:04:23.848 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:04:23.905 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:04:23.906 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:04:24.055 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:04:24.122 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:04:24.146 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:04:24.147 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:24.147 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:04:24.147 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:25.096 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:04:25.096 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:25.096 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:04:25.096 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:25.096 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:25.096 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:04:25.362 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr 2024-03-25T18:04:25.932 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:04:26.036 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:04:26.091 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:04:26.093 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-25T18:04:26.093 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:04:26.158 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:04:26.729 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:04:42 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:04:26.884 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:04:26.886 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================== 2024-03-25T18:04:26.886 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-25T18:04:26.886 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================== 2024-03-25T18:04:26.886 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:04:26.886 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:04:26.886 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-25T18:04:26.886 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:04:26.886 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:04:26.886 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:04:26.886 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================== 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2024-03-25T18:04:26.887 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:26.888 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 6.9 M 2024-03-25T18:04:26.888 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 26 M 2024-03-25T18:04:26.888 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:04:27.774 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 29 kB/s | 15 kB 00:00 2024-03-25T18:04:27.800 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 56 kB/s | 31 kB 00:00 2024-03-25T18:04:27.941 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-25T18:04:27.975 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-25T18:04:28.259 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-03-25T18:04:28.534 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-25T18:04:28.568 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 925 kB/s | 548 kB 00:00 2024-03-25T18:04:28.993 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 340 kB/s | 144 kB 00:00 2024-03-25T18:04:29.260 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 115 kB/s | 31 kB 00:00 2024-03-25T18:04:29.302 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.7 MB/s | 2.0 MB 00:00 2024-03-25T18:04:29.685 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 443 kB/s | 188 kB 00:00 2024-03-25T18:04:32.972 INFO:teuthology.orchestra.run.smithi179.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 494 kB/s | 17 MB 00:34 2024-03-25T18:04:32.977 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:32.977 INFO:teuthology.orchestra.run.smithi179.stdout:Total 774 kB/s | 30 MB 00:39 2024-03-25T18:04:32.999 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:04:33.042 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:04:33.042 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:04:33.452 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:04:33.453 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:04:33.885 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 274 kB/s | 1.8 MB 00:06 2024-03-25T18:04:33.890 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:33.890 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.0 MB/s | 6.9 MB 00:07 2024-03-25T18:04:33.910 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:04:33.930 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:04:33.930 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:04:34.114 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:04:34.114 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:04:34.652 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:04:34.720 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:04:34.755 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:04:34.778 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:04:34.802 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:04:34.839 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:04:34.841 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:04:34.885 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:04:34.910 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:04:34.938 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:04:34.980 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:04:35.025 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:04:35.058 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:04:35.089 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:04:35.149 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:04:35.236 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:04:35.271 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:04:35.277 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:04:35.319 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:04:35.347 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:04:35.348 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:04:35.379 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:04:35.403 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:04:35.450 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:04:35.499 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:04:35.553 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:04:35.634 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:04:35.662 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:04:35.738 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:04:35.780 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:04:35.818 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:04:35.852 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:04:35.874 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:04:35.950 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:04:35.985 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:04:36.018 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:04:36.058 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:04:36.093 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:04:36.128 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:04:36.216 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:04:36.250 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:04:36.299 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:04:36.299 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:04:36.300 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:04:36.300 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:04:36.300 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:04:36.300 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:04:36.300 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:04:36.300 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:04:36.300 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:04:36.300 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:04:36.300 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:04:36.300 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:04:36.332 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:04:36.468 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:04:36.593 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:04:36.663 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:04:36.664 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:04:36.665 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:04:36.665 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:36.665 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:04:36.753 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:04:36.782 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:04:36.806 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:04:36.806 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:36.806 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:04:36.806 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:04:36.806 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:04:36.807 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:36.916 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:04:37.484 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:04:53 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:04:37.641 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:04:37.642 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-03-25T18:04:37.642 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-25T18:04:37.642 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-03-25T18:04:37.642 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:04:37.643 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:04:37.644 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:04:37.644 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:37.644 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:04:37.644 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2024-03-25T18:04:37.644 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2024-03-25T18:04:37.644 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:04:37.644 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 37 M 2024-03-25T18:04:37.645 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 209 M 2024-03-25T18:04:37.645 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:04:37.908 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 297 kB/s | 32 kB 00:00 2024-03-25T18:04:37.957 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 669 kB/s | 17 kB 00:00 2024-03-25T18:04:37.991 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:04:38.166 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T18:04:38.300 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:04:38.317 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-25T18:04:38.327 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:04:38.327 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:38.327 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:04:38.327 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:04:38.327 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:04:38.327 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:38.375 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-25T18:04:38.401 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-25T18:04:38.576 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-25T18:04:38.626 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 195 kB 00:00 2024-03-25T18:04:38.768 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:04:38.785 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-25T18:04:38.790 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:04:38.790 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:38.790 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:04:38.791 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:04:38.791 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:04:38.791 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:38.810 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-25T18:04:39.001 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:04:39.023 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:04:39.024 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:39.024 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:04:39.024 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:04:39.024 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:04:39.024 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:39.090 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:04:40.832 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:04:40.833 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:04:40.834 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:04:40.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:04:40.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:04:40.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:04:40.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:04:40.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:04:40.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:04:40.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:04:40.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:04:40.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:04:40.836 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:04:41.960 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:04:41.960 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:41.960 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:04:41.960 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:41.960 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:04:41.961 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:04:41.962 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:04:41.962 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:04:41.962 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:04:41.962 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:04:41.962 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:04:41.962 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:04:41.962 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:04:41.962 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:04:41.965 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:04:41.965 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:04:41.965 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:04:41.966 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:04:41.966 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:04:41.966 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:04:41.966 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:04:41.966 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:04:41.966 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:04:41.966 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:04:41.966 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:04:41.966 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:04:41.966 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:41.966 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:04:42.306 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-base 2024-03-25T18:04:42.876 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:04:42.973 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:04:43.025 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:04:43.026 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-25T18:04:43.026 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:04:43.092 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install cephadm 2024-03-25T18:04:43.655 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:05:03 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:04:43.806 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:04:43.806 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:04:43.806 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-25T18:04:43.806 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:04:43.806 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:04:43.807 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:04:43.807 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:43.807 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:04:43.807 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:04:43.807 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T18:04:43.807 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:43.807 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 747 k 2024-03-25T18:04:43.807 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 754 k 2024-03-25T18:04:43.807 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:04:47.734 INFO:teuthology.orchestra.run.smithi179.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 190 kB/s | 747 kB 00:03 2024-03-25T18:04:47.735 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:47.735 INFO:teuthology.orchestra.run.smithi179.stdout:Total 190 kB/s | 747 kB 00:03 2024-03-25T18:04:47.736 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:04:47.741 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:04:47.741 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:04:47.747 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:04:47.748 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:04:47.825 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:04:48.292 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:48.388 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:48.783 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:49.081 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:04:49.081 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:49.081 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:04:49.081 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:04:49.081 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:49.081 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:04:49.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:04:49.868 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:04:50.023 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:04:50.023 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 145 k 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 436 k 2024-03-25T18:04:50.024 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:04:51.181 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 126 kB/s | 145 kB 00:01 2024-03-25T18:04:51.182 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:51.183 INFO:teuthology.orchestra.run.smithi179.stdout:Total 125 kB/s | 145 kB 00:01 2024-03-25T18:04:51.183 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:04:51.191 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:04:51.192 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:04:51.248 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:04:51.248 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:04:51.398 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:04:51.449 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:04:51.474 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:04:51.474 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:04:51.474 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:04:51.474 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:52.531 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:04:52.531 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:52.531 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:04:52.531 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:52.531 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:52.531 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:04:52.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr 2024-03-25T18:04:53.354 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:04:53.452 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:04:53.506 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:04:53.506 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-25T18:04:53.506 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:04:53.563 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:04:54.117 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:04:54.267 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:04:54.268 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-25T18:04:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-25T18:04:54.268 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-25T18:04:54.268 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:04:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:04:54.268 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-25T18:04:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:04:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:04:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:04:54.268 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout:Install 12 Packages 2024-03-25T18:04:54.269 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:04:54.270 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 6.9 M 2024-03-25T18:04:54.270 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 26 M 2024-03-25T18:04:54.270 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:04:54.843 INFO:teuthology.orchestra.run.smithi179.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 33 kB/s | 15 kB 00:00 2024-03-25T18:04:54.910 INFO:teuthology.orchestra.run.smithi179.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 59 kB/s | 31 kB 00:00 2024-03-25T18:04:54.977 INFO:teuthology.orchestra.run.smithi179.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.5 MB/s | 238 kB 00:00 2024-03-25T18:04:55.027 INFO:teuthology.orchestra.run.smithi179.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-25T18:04:55.270 INFO:teuthology.orchestra.run.smithi179.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.9 MB/s | 863 kB 00:00 2024-03-25T18:04:55.486 INFO:teuthology.orchestra.run.smithi179.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T18:04:55.512 INFO:teuthology.orchestra.run.smithi179.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-25T18:04:55.712 INFO:teuthology.orchestra.run.smithi179.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 722 kB/s | 144 kB 00:00 2024-03-25T18:04:55.829 INFO:teuthology.orchestra.run.smithi179.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 6.0 MB/s | 2.0 MB 00:00 2024-03-25T18:04:55.854 INFO:teuthology.orchestra.run.smithi179.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 216 kB/s | 31 kB 00:00 2024-03-25T18:04:55.929 INFO:teuthology.orchestra.run.smithi179.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-25T18:05:00.413 INFO:teuthology.orchestra.run.smithi179.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 302 kB/s | 1.8 MB 00:06 2024-03-25T18:05:00.419 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:00.419 INFO:teuthology.orchestra.run.smithi179.stdout:Total 1.1 MB/s | 6.9 MB 00:06 2024-03-25T18:05:00.439 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:05:00.459 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:05:00.459 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:05:00.630 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:05:00.630 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:05:01.144 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:05:01.229 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:05:01.265 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:05:01.312 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:05:01.342 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:05:01.443 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:05:01.575 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:05:01.670 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:05:01.713 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:05:01.742 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:05:01.774 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:05:02.015 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:05:02.049 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:05:02.199 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 310 kB/s | 7.4 MB 00:24 2024-03-25T18:05:02.202 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:02.203 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.5 MB/s | 37 MB 00:24 2024-03-25T18:05:02.244 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:05:02.271 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:05:02.272 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:05:02.645 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:05:02.646 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:05:02.815 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:05:02.815 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:05:02.815 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:05:02.815 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:05:02.816 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:05:02.816 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:05:02.816 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:05:02.816 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:05:02.816 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:05:02.816 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:05:02.816 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:05:02.816 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:05:03.171 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:05:03.172 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:05:03.173 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:03.173 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:05:03.371 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:05:03.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:05:03.456 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:05:03.487 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:05:03.586 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:05:03.624 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:05:03.816 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:05:03.856 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:05:03.943 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:05:03.976 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:05:23 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:05:04.125 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:05:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-25T18:05:04.126 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-25T18:05:04.126 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-25T18:05:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:05:04.127 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:05:04.127 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-25T18:05:04.127 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:05:04.127 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:05:04.127 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:05:04.127 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:05:04.127 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:05:04.127 INFO:teuthology.orchestra.run.smithi179.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:05:04.127 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:05:04.127 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:05:04.128 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:05:04.128 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:05:04.128 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:05:04.128 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:05:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-25T18:05:04.128 INFO:teuthology.orchestra.run.smithi179.stdout:Install 12 Packages 2024-03-25T18:05:04.128 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 37 M 2024-03-25T18:05:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 209 M 2024-03-25T18:05:04.129 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:05:04.419 INFO:teuthology.orchestra.run.smithi179.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 278 kB/s | 32 kB 00:00 2024-03-25T18:05:04.478 INFO:teuthology.orchestra.run.smithi179.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 485 kB/s | 17 kB 00:00 2024-03-25T18:05:04.511 INFO:teuthology.orchestra.run.smithi179.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:05:04.639 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:05:04.695 INFO:teuthology.orchestra.run.smithi179.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-25T18:05:04.756 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:05:04.837 INFO:teuthology.orchestra.run.smithi179.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T18:05:04.904 INFO:teuthology.orchestra.run.smithi179.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2024-03-25T18:05:04.929 INFO:teuthology.orchestra.run.smithi179.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-25T18:05:05.238 INFO:teuthology.orchestra.run.smithi179.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-25T18:05:05.313 INFO:teuthology.orchestra.run.smithi179.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-25T18:05:05.413 INFO:teuthology.orchestra.run.smithi179.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-25T18:05:05.455 INFO:teuthology.orchestra.run.smithi179.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-25T18:05:06.079 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:05:07.162 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:05:07.220 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:05:09.526 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:05:09.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:05:09.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:05:09.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:05:09.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:05:09.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:05:09.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:05:09.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:05:09.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:05:09.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:05:09.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:05:09.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:05:10.027 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:05:10.027 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:10.027 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:05:10.027 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:10.027 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:10.028 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:10.375 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:05:10.950 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:26 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:11.107 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:11.108 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:05:11.109 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:05:11.110 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:05:11.110 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:05:11.110 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:05:11.110 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:11.110 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:05:11.110 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-25T18:05:11.110 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2024-03-25T18:05:11.110 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:11.110 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.0 M 2024-03-25T18:05:11.111 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 26 M 2024-03-25T18:05:11.111 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:05:11.437 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 180 kB/s | 24 kB 00:00 2024-03-25T18:05:11.462 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 112 kB/s | 18 kB 00:00 2024-03-25T18:05:11.520 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2024-03-25T18:05:11.537 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T18:05:11.562 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-25T18:05:11.580 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T18:05:11.813 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 129 kB/s | 32 kB 00:00 2024-03-25T18:05:11.830 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-25T18:05:11.855 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 94 kB/s | 52 kB 00:00 2024-03-25T18:05:11.997 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.1 MB/s | 203 kB 00:00 2024-03-25T18:05:12.031 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-03-25T18:05:12.131 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.4 MB/s | 1.0 MB 00:00 2024-03-25T18:05:12.156 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-03-25T18:05:12.160 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:12.160 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-25T18:05:12.160 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:05:12.175 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:05:12.175 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:05:12.272 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:05:12.274 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:05:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:05:13.112 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:05:13.168 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:05:13.223 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:05:13.259 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:05:13.301 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:05:13.399 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:05:13.508 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:05:13.569 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:05:14.332 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:05:14.373 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:05:14.407 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:05:14.456 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:05:14.484 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:05:15.277 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:05:15.277 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:05:15.277 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:05:15.277 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:05:15.277 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:05:15.277 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:05:15.277 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:05:15.277 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:05:15.277 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:05:15.277 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:05:15.277 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:05:15.278 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:05:15.278 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:05:15.642 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:05:15.642 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:15.642 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:05:15.642 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:15.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:05:15.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:05:15.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:05:15.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:05:15.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:05:15.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:05:15.643 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:05:15.643 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:05:15.643 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:05:15.643 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:05:15.643 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:05:15.643 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:05:15.643 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:15.643 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:15.877 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:05:16.452 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:32 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:16.609 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:16.610 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-25T18:05:16.610 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-25T18:05:16.610 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-25T18:05:16.610 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:05:16.610 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:05:16.610 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-25T18:05:16.610 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:05:16.611 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:05:16.611 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:05:16.611 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:05:16.611 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:05:16.611 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:16.611 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:05:16.611 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-25T18:05:16.611 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-03-25T18:05:16.611 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:16.611 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 6.8 M 2024-03-25T18:05:16.612 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 33 M 2024-03-25T18:05:16.612 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:05:17.040 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-25T18:05:17.132 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T18:05:17.316 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 363 kB/s | 58 kB 00:00 2024-03-25T18:05:17.374 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-25T18:05:17.391 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 682 kB/s | 51 kB 00:00 2024-03-25T18:05:17.866 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 138 kB/s | 138 kB 00:00 2024-03-25T18:05:17.870 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:17.870 INFO:teuthology.orchestra.run.smithi052.stdout:Total 5.4 MB/s | 6.8 MB 00:01 2024-03-25T18:05:17.884 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:05:17.896 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:05:17.896 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:05:17.990 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:05:17.990 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:05:18.300 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:05:18.463 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:05:18.519 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:05:19.092 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:05:19.173 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:05:19.234 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:05:19.272 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:05:20.056 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:05:20.056 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:05:20.056 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:05:20.056 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:05:20.056 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:05:20.056 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:05:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:05:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:20.363 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:05:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:05:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:05:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:05:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:05:20.363 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:05:20.364 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:20.364 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:20.596 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse 2024-03-25T18:05:21.169 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:37 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:21.324 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:21.326 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 964 k 2024-03-25T18:05:21.327 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.8 M 2024-03-25T18:05:21.327 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:05:21.468 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 558 kB/s | 79 kB 00:00 2024-03-25T18:05:26.376 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 175 kB/s | 885 kB 00:05 2024-03-25T18:05:26.378 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:26.378 INFO:teuthology.orchestra.run.smithi052.stdout:Total 191 kB/s | 964 kB 00:05 2024-03-25T18:05:26.378 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:05:26.404 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:05:26.404 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:05:26.484 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:05:26.484 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:05:26.671 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:05:26.800 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:05:26.889 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:05:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:05:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:05:27.888 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:05:27.888 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:27.888 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:05:27.888 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:05:27.888 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:27.889 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:28.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-volume 2024-03-25T18:05:28.729 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:44 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:28.885 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:28.886 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:28.886 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-03-25T18:05:28.886 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:28.886 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:05:28.886 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:05:28.886 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-25T18:05:28.886 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:05:28.886 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:28.886 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:05:28.887 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:28.887 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-03-25T18:05:28.887 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:28.887 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 332 k 2024-03-25T18:05:28.887 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.5 M 2024-03-25T18:05:28.887 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:05:28.998 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 622 kB/s | 69 kB 00:00 2024-03-25T18:05:30.748 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 142 kB/s | 264 kB 00:01 2024-03-25T18:05:30.749 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:30.750 INFO:teuthology.orchestra.run.smithi052.stdout:Total 178 kB/s | 332 kB 00:01 2024-03-25T18:05:30.750 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:05:30.760 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:05:30.761 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:05:30.807 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:05:30.807 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:05:30.958 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:05:31.145 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:05:31.205 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:05:31.221 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:05:31.221 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:31.221 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:05:31.221 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:32.124 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:05:32.469 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:05:32.469 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:32.469 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:05:32.469 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:32.469 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:05:32.469 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:32.469 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:32.820 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel 2024-03-25T18:05:33.400 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:49 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:33.556 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:33.558 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:33.558 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T18:05:33.558 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:33.558 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:05:33.558 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:05:33.558 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:33.558 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:05:33.558 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:33.559 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:05:33.559 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:33.559 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 127 k 2024-03-25T18:05:33.559 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 456 k 2024-03-25T18:05:33.559 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:05:34.245 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 184 kB/s | 127 kB 00:00 2024-03-25T18:05:34.246 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:34.246 INFO:teuthology.orchestra.run.smithi052.stdout:Total 184 kB/s | 127 kB 00:00 2024-03-25T18:05:34.246 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:05:34.252 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:05:34.253 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:05:34.314 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:05:34.314 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:05:34.462 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:05:34.572 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:35.053 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:35.460 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:35.460 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:35.460 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:05:35.460 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:35.460 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:35.460 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:35.683 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2 2024-03-25T18:05:36.261 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:52 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:36.362 INFO:teuthology.orchestra.run.smithi052.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:36.418 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:36.420 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-25T18:05:36.420 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:36.491 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs-devel 2024-03-25T18:05:37.064 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:53 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:37.219 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:37.220 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:37.220 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T18:05:37.220 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:37.220 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:05:37.220 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:05:37.221 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:37.221 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:05:37.221 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:37.221 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:05:37.221 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:37.221 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 31 k 2024-03-25T18:05:37.221 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 140 k 2024-03-25T18:05:37.221 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:05:37.756 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 58 kB/s | 31 kB 00:00 2024-03-25T18:05:37.757 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:37.757 INFO:teuthology.orchestra.run.smithi052.stdout:Total 58 kB/s | 31 kB 00:00 2024-03-25T18:05:37.757 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:05:37.761 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:05:37.761 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:05:37.787 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:05:37.787 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:05:37.907 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:05:38.007 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:38.380 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:38.661 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:38.661 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:38.661 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:05:38.661 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:38.661 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:38.661 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:38.823 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2 2024-03-25T18:05:39.400 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:55 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:39.503 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:39.559 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:39.560 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-25T18:05:39.561 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:39.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1 2024-03-25T18:05:39.627 INFO:teuthology.orchestra.run.smithi179.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 214 kB/s | 7.4 MB 00:35 2024-03-25T18:05:39.629 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:39.629 INFO:teuthology.orchestra.run.smithi179.stdout:Total 1.0 MB/s | 37 MB 00:35 2024-03-25T18:05:39.671 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:05:39.697 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:05:39.697 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:05:40.058 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:05:40.059 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:05:40.271 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:56 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:40.372 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:40.426 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:40.428 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-25T18:05:40.428 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:40.471 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados 2024-03-25T18:05:40.758 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:05:40.825 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:05:40.855 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:05:40.948 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:05:41.000 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:05:41.041 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:57 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:41.141 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:41.169 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:05:41.196 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:41.197 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-25T18:05:41.197 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:41.199 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:05:41.251 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw 2024-03-25T18:05:41.285 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:05:41.821 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:57 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:41.922 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:41.932 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:05:41.976 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:41.978 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-25T18:05:41.978 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:42.025 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs 2024-03-25T18:05:42.066 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:05:42.596 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:58 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:42.695 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:42.751 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:42.752 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-25T18:05:42.752 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:42.797 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd 2024-03-25T18:05:43.290 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:05:43.366 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:05:59 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:43.466 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:43.520 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:43.522 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-25T18:05:43.522 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:43.563 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse 2024-03-25T18:05:44.135 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:06:00 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:44.294 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:44.295 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:44.295 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-03-25T18:05:44.295 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:44.295 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:05:44.295 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:05:44.295 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:44.295 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:05:44.295 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:44.295 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:05:44.295 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:44.295 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 85 k 2024-03-25T18:05:44.296 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 231 k 2024-03-25T18:05:44.296 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:05:44.328 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:05:44.357 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:05:45.085 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 108 kB/s | 85 kB 00:00 2024-03-25T18:05:45.085 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:45.085 INFO:teuthology.orchestra.run.smithi052.stdout:Total 108 kB/s | 85 kB 00:00 2024-03-25T18:05:45.086 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:05:45.093 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:05:45.093 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:05:45.154 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:05:45.155 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:05:45.301 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:05:45.436 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:45.832 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:46.134 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:46.134 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:46.134 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:05:46.134 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:46.135 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:46.135 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:05:46.343 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2024-03-25T18:05:46.917 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:06:02 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:05:46.929 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:05:46.929 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:05:46.930 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:05:46.930 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:05:46.930 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:05:46.930 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:05:46.930 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:05:46.930 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:05:46.930 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:05:46.930 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:05:46.930 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:05:46.930 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:05:47.073 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:05:47.074 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:47.074 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T18:05:47.074 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:47.074 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:05:47.074 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:05:47.074 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:47.074 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:05:47.074 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:05:47.074 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:05:47.074 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:47.075 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.1 M 2024-03-25T18:05:47.075 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 13 M 2024-03-25T18:05:47.075 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:05:47.402 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:05:47.402 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:47.402 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:05:47.402 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:47.402 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:05:47.402 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:05:47.402 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:05:47.402 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:05:47.403 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:05:47.403 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:05:47.403 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:05:47.403 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:05:47.403 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:05:47.403 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:05:47.403 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:05:47.403 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:47.403 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:05:47.786 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:05:48.360 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:08 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:05:48.510 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:05:48.511 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-25T18:05:48.511 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-25T18:05:48.511 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:05:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:05:48.513 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:05:48.513 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:48.513 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:05:48.513 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-25T18:05:48.513 INFO:teuthology.orchestra.run.smithi179.stdout:Install 13 Packages 2024-03-25T18:05:48.513 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:48.513 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 2.0 M 2024-03-25T18:05:48.513 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 26 M 2024-03-25T18:05:48.514 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:05:48.660 INFO:teuthology.orchestra.run.smithi179.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 240 kB/s | 24 kB 00:00 2024-03-25T18:05:48.701 INFO:teuthology.orchestra.run.smithi179.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 18 kB 00:00 2024-03-25T18:05:48.744 INFO:teuthology.orchestra.run.smithi179.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.8 MB/s | 174 kB 00:00 2024-03-25T18:05:48.769 INFO:teuthology.orchestra.run.smithi179.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T18:05:48.794 INFO:teuthology.orchestra.run.smithi179.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:05:48.811 INFO:teuthology.orchestra.run.smithi179.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T18:05:48.928 INFO:teuthology.orchestra.run.smithi179.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-25T18:05:48.945 INFO:teuthology.orchestra.run.smithi179.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 215 kB/s | 32 kB 00:00 2024-03-25T18:05:49.154 INFO:teuthology.orchestra.run.smithi179.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 899 kB/s | 203 kB 00:00 2024-03-25T18:05:49.204 INFO:teuthology.orchestra.run.smithi179.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-25T18:05:49.262 INFO:teuthology.orchestra.run.smithi179.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.2 MB/s | 1.0 MB 00:00 2024-03-25T18:05:49.279 INFO:teuthology.orchestra.run.smithi179.stdout:(12/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 72 kB/s | 52 kB 00:00 2024-03-25T18:05:49.304 INFO:teuthology.orchestra.run.smithi179.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 893 kB/s | 90 kB 00:00 2024-03-25T18:05:49.307 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:49.307 INFO:teuthology.orchestra.run.smithi179.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-25T18:05:49.307 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:05:49.320 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:05:49.320 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:05:49.409 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:05:49.410 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:05:49.918 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:05:50.083 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:05:50.139 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:05:50.194 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:05:50.231 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:05:50.273 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:05:50.366 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:05:50.466 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:05:50.524 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:05:51.241 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:05:51.288 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:05:51.322 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:05:51.370 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:05:51.392 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:05:52.126 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:05:52.127 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:05:52.498 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:05:52.498 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:52.498 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:05:52.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:52.498 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:05:52.498 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:05:52.498 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:05:52.498 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:05:52.498 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:05:52.498 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:05:52.499 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:05:52.499 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:05:52.499 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:05:52.499 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:05:52.499 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:05:52.499 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:05:52.499 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:52.499 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:05:52.761 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:05:53.325 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:13 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:05:53.475 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:05:53.477 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-25T18:05:53.477 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-25T18:05:53.477 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-25T18:05:53.477 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:05:53.477 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:05:53.477 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-25T18:05:53.477 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:05:53.477 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:05:53.477 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:05:53.477 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:05:53.477 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:05:53.478 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:53.478 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:05:53.478 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-25T18:05:53.478 INFO:teuthology.orchestra.run.smithi179.stdout:Install 6 Packages 2024-03-25T18:05:53.478 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:53.478 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 6.8 M 2024-03-25T18:05:53.478 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 33 M 2024-03-25T18:05:53.478 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:05:53.981 INFO:teuthology.orchestra.run.smithi179.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T18:05:54.306 INFO:teuthology.orchestra.run.smithi179.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 416 kB/s | 248 kB 00:00 2024-03-25T18:05:54.398 INFO:teuthology.orchestra.run.smithi179.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T18:05:54.423 INFO:teuthology.orchestra.run.smithi179.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 494 kB/s | 58 kB 00:00 2024-03-25T18:05:54.465 INFO:teuthology.orchestra.run.smithi179.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 762 kB/s | 51 kB 00:00 2024-03-25T18:05:54.657 INFO:teuthology.orchestra.run.smithi179.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 146 kB/s | 138 kB 00:00 2024-03-25T18:05:54.659 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:54.660 INFO:teuthology.orchestra.run.smithi179.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-25T18:05:54.673 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:05:54.683 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:05:54.683 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:05:54.771 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:05:54.773 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:05:55.073 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:05:55.238 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:05:55.283 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:05:55.798 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:05:55.888 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:05:55.949 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:05:55.979 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:05:56.843 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:05:56.843 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:05:56.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:05:56.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:05:56.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:05:56.844 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:05:57.147 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:05:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:57.148 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:05:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:05:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:05:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:05:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:05:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:05:57.148 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:57.148 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:05:57.413 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-fuse 2024-03-25T18:05:57.973 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:05:58.061 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 288 kB/s | 3.1 MB 00:10 2024-03-25T18:05:58.062 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:58.062 INFO:teuthology.orchestra.run.smithi052.stdout:Total 288 kB/s | 3.1 MB 00:10 2024-03-25T18:05:58.063 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:05:58.074 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:05:58.074 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:05:58.121 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:05:58.122 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:05:58.122 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-25T18:05:58.122 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:05:58.122 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:05:58.122 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:05:58.122 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-25T18:05:58.122 INFO:teuthology.orchestra.run.smithi179.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:05:58.122 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:58.122 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:05:58.123 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:05:58.123 INFO:teuthology.orchestra.run.smithi179.stdout:Install 2 Packages 2024-03-25T18:05:58.123 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:05:58.123 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 964 k 2024-03-25T18:05:58.123 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 2.8 M 2024-03-25T18:05:58.123 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:05:58.154 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:05:58.155 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:05:58.242 INFO:teuthology.orchestra.run.smithi179.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 661 kB/s | 79 kB 00:00 2024-03-25T18:05:58.603 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:05:58.670 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:58.694 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:58.694 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:58.694 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:05:58.694 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:05:58.694 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:05:58.694 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:59.908 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:59.908 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:59.908 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:05:59.908 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:59.908 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:05:59.908 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:06:00.148 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd 2024-03-25T18:06:00.729 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:06:16 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:06:00.883 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:06:00.884 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:06:00.884 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-03-25T18:06:00.884 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:06:00.884 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:06:00.884 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:06:00.884 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:06:00.884 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:06:00.884 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:06:00.884 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:06:00.884 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:06:00.885 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 171 k 2024-03-25T18:06:00.885 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 494 k 2024-03-25T18:06:00.885 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:06:01.201 INFO:teuthology.orchestra.run.smithi179.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 288 kB/s | 885 kB 00:03 2024-03-25T18:06:01.203 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:01.203 INFO:teuthology.orchestra.run.smithi179.stdout:Total 313 kB/s | 964 kB 00:03 2024-03-25T18:06:01.203 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:06:01.227 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:06:01.227 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:06:01.305 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:06:01.305 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:06:01.494 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:06:01.643 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:06:01.717 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:06:01.730 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 203 kB/s | 171 kB 00:00 2024-03-25T18:06:01.730 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:01.731 INFO:teuthology.orchestra.run.smithi052.stdout:Total 203 kB/s | 171 kB 00:00 2024-03-25T18:06:01.731 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:06:01.740 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:06:01.741 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:06:01.804 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:06:01.804 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:06:01.959 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:06:02.056 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:02.392 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:06:02.393 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:06:02.482 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:02.690 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:06:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:02.691 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:06:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:06:02.691 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:02.691 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:02.815 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:02.815 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:06:02.815 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:06:02.815 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:02.815 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:06:02.816 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:06:02.929 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-volume 2024-03-25T18:06:03.039 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:06:03.500 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:23 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:03.615 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:06:19 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:06:03.654 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:03.655 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:03.655 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-25T18:06:03.655 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:03.655 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:06:03.656 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:06:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-25T18:06:03.656 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:06:03.656 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:06:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:03.656 INFO:teuthology.orchestra.run.smithi179.stdout:Install 2 Packages 2024-03-25T18:06:03.656 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:03.657 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 332 k 2024-03-25T18:06:03.657 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 1.5 M 2024-03-25T18:06:03.657 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:06:03.763 INFO:teuthology.orchestra.run.smithi179.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 645 kB/s | 69 kB 00:00 2024-03-25T18:06:03.770 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:06:03.771 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:06:03.771 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T18:06:03.771 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:06:03.771 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:06:03.771 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:06:03.771 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-03-25T18:06:03.771 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:06:03.771 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:06:03.771 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:06:03.771 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:06:03.772 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-03-25T18:06:03.772 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:06:03.772 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 91 M 2024-03-25T18:06:03.772 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 498 M 2024-03-25T18:06:03.772 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:06:05.971 INFO:teuthology.orchestra.run.smithi179.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 114 kB/s | 264 kB 00:02 2024-03-25T18:06:05.973 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:05.973 INFO:teuthology.orchestra.run.smithi179.stdout:Total 143 kB/s | 332 kB 00:02 2024-03-25T18:06:05.973 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:06:05.983 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:06:05.983 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:06:06.029 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:06:06.030 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:06:06.184 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:06:06.379 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:06:06.415 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:06:06.430 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:06:06.431 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:06.431 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:06:06.431 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:07.150 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:06:07.494 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:06:07.495 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:07.495 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:06:07.495 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:07.495 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:06:07.495 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:07.495 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:07.810 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados-devel 2024-03-25T18:06:08.382 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:28 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:08.530 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:08.530 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:08.530 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T18:06:08.530 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:08.530 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:06:08.530 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:06:08.531 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:08.531 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:06:08.531 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:08.531 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T18:06:08.531 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:08.531 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 127 k 2024-03-25T18:06:08.531 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 456 k 2024-03-25T18:06:08.532 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:06:09.211 INFO:teuthology.orchestra.run.smithi179.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 186 kB/s | 127 kB 00:00 2024-03-25T18:06:09.211 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:09.211 INFO:teuthology.orchestra.run.smithi179.stdout:Total 186 kB/s | 127 kB 00:00 2024-03-25T18:06:09.212 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:06:09.217 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:06:09.217 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:06:09.274 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:06:09.275 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:06:09.418 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:06:09.526 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:09.982 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:10.261 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:06:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:10.261 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:10.261 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:10.479 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs2 2024-03-25T18:06:11.047 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:31 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:11.146 INFO:teuthology.orchestra.run.smithi179.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:11.201 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:11.202 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-25T18:06:11.202 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:11.256 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs-devel 2024-03-25T18:06:11.818 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:31 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:11.968 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:11.969 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:11.969 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T18:06:11.969 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:11.969 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:06:11.969 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:06:11.969 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:11.969 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:06:11.969 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:11.969 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T18:06:11.969 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:11.970 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 31 k 2024-03-25T18:06:11.970 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 140 k 2024-03-25T18:06:11.970 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:06:12.656 INFO:teuthology.orchestra.run.smithi179.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 45 kB/s | 31 kB 00:00 2024-03-25T18:06:12.657 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:12.657 INFO:teuthology.orchestra.run.smithi179.stdout:Total 45 kB/s | 31 kB 00:00 2024-03-25T18:06:12.657 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:06:12.660 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:06:12.660 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:06:12.686 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:06:12.686 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:06:12.807 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:06:12.912 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:13.295 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:13.560 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:13.560 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:13.560 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:06:13.560 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:13.560 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:13.560 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:13.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados2 2024-03-25T18:06:14.275 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:34 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:14.377 INFO:teuthology.orchestra.run.smithi179.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:14.434 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:14.434 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-25T18:06:14.434 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:14.486 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librbd1 2024-03-25T18:06:15.078 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:35 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:15.181 INFO:teuthology.orchestra.run.smithi179.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:15.235 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:15.236 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-25T18:06:15.236 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:15.283 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rados 2024-03-25T18:06:15.847 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:35 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:15.945 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:15.996 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:15.997 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-25T18:06:15.997 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:16.042 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rgw 2024-03-25T18:06:16.601 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:36 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:16.697 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:16.748 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:16.749 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-25T18:06:16.749 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:16.798 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-cephfs 2024-03-25T18:06:17.360 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:37 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:17.456 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:17.507 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:17.508 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-25T18:06:17.508 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:17.566 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rbd 2024-03-25T18:06:18.126 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:38 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:18.223 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:18.276 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:18.277 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-25T18:06:18.277 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:18.322 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-fuse 2024-03-25T18:06:18.883 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:38 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:19.032 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:19.032 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:19.032 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-03-25T18:06:19.032 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:19.032 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:06:19.032 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:06:19.033 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:19.033 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:06:19.033 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:19.033 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T18:06:19.033 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:19.033 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 85 k 2024-03-25T18:06:19.033 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 231 k 2024-03-25T18:06:19.033 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:06:19.743 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 120 kB/s | 85 kB 00:00 2024-03-25T18:06:19.743 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:19.744 INFO:teuthology.orchestra.run.smithi179.stdout:Total 120 kB/s | 85 kB 00:00 2024-03-25T18:06:19.744 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:06:19.751 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:06:19.751 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:06:19.808 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:06:19.808 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:06:19.943 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:06:20.050 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:20.492 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:20.765 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:20.765 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:20.765 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:06:20.765 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:20.765 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:20.765 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:20.993 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-mirror 2024-03-25T18:06:21.562 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:41 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:21.715 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:21.716 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:21.716 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T18:06:21.716 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:21.716 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:06:21.716 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:06:21.716 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:21.717 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:06:21.717 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:21.717 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T18:06:21.717 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:21.717 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 3.1 M 2024-03-25T18:06:21.717 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 13 M 2024-03-25T18:06:21.717 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:06:32.970 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 281 kB/s | 3.1 MB 00:11 2024-03-25T18:06:32.970 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:32.971 INFO:teuthology.orchestra.run.smithi179.stdout:Total 281 kB/s | 3.1 MB 00:11 2024-03-25T18:06:32.971 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:06:32.982 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:06:32.982 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:06:33.058 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:06:33.058 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:06:33.484 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:06:33.554 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:33.577 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:33.577 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:33.577 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:06:33.577 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:06:33.577 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:06:33.577 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:34.708 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:34.709 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:34.709 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:06:34.709 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:34.709 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:34.709 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:34.957 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-nbd 2024-03-25T18:06:35.533 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:55 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:35.682 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:35.683 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:35.683 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-03-25T18:06:35.683 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:35.683 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:06:35.683 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:06:35.683 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:35.683 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:06:35.684 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:35.684 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T18:06:35.684 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:35.684 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 171 k 2024-03-25T18:06:35.684 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 494 k 2024-03-25T18:06:35.684 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:06:37.056 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 125 kB/s | 171 kB 00:01 2024-03-25T18:06:37.057 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:37.057 INFO:teuthology.orchestra.run.smithi179.stdout:Total 125 kB/s | 171 kB 00:01 2024-03-25T18:06:37.057 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:06:37.066 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:06:37.067 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:06:37.127 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:06:37.127 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:06:37.283 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:06:37.371 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:37.869 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:38.149 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:38.149 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:38.149 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:06:38.149 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:38.149 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:38.149 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:06:38.399 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:06:38.967 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:06:58 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:06:39.122 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:06:39.123 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:39.123 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T18:06:39.123 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:39.123 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:06:39.123 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:06:39.123 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-25T18:06:39.123 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:06:39.124 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:39.124 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:06:39.124 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:06:39.124 INFO:teuthology.orchestra.run.smithi179.stdout:Install 2 Packages 2024-03-25T18:06:39.124 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:06:39.124 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 91 M 2024-03-25T18:06:39.124 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 498 M 2024-03-25T18:06:39.125 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:07:11.118 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 341 kB/s | 22 MB 01:07 2024-03-25T18:07:49.127 INFO:teuthology.orchestra.run.smithi179.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 328 kB/s | 22 MB 01:10 2024-03-25T18:08:40.158 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 446 kB/s | 68 MB 02:36 2024-03-25T18:08:40.159 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:40.160 INFO:teuthology.orchestra.run.smithi052.stdout:Total 593 kB/s | 91 MB 02:36 2024-03-25T18:08:40.160 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:08:40.164 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:08:40.164 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:08:40.778 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:08:40.779 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:08:44.054 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:08:51.551 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:08:51.636 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:08:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:08:54.863 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:08:55.138 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:08:55.139 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:08:55.139 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:08:55.139 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:55.139 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:55.139 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:08:55.139 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:08:55.352 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:08:55.925 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:09:11 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:08:56.086 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:08:56.087 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:08:56.087 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T18:08:56.087 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:08:56.088 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:08:56.088 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:08:56.088 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:08:56.088 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:08:56.088 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:08:56.088 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:08:56.088 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:08:56.088 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 416 M 2024-03-25T18:08:56.088 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.7 G 2024-03-25T18:08:56.088 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:09:32.026 INFO:teuthology.orchestra.run.smithi179.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 404 kB/s | 68 MB 02:52 2024-03-25T18:09:32.027 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:32.027 INFO:teuthology.orchestra.run.smithi179.stdout:Total 536 kB/s | 91 MB 02:52 2024-03-25T18:09:32.027 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:09:32.028 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:09:32.028 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:09:32.611 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:09:32.612 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:09:35.767 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:09:42.822 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:09:42.890 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:09:46.346 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:09:46.346 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:09:46.623 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:09:46.623 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:09:46.623 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:09:46.623 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:46.623 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:46.623 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:09:46.623 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:09:46.823 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:09:47.384 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:10:07 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:09:47.530 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:09:47.531 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:09:47.531 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T18:09:47.531 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:09:47.531 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:09:47.531 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:09:47.531 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:09:47.531 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:09:47.531 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:09:47.531 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T18:09:47.531 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:09:47.532 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 416 M 2024-03-25T18:09:47.532 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 1.7 G 2024-03-25T18:09:47.532 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:24:21.206 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 461 kB/s | 416 MB 15:25 2024-03-25T18:24:21.207 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:21.207 INFO:teuthology.orchestra.run.smithi052.stdout:Total 461 kB/s | 416 MB 15:25 2024-03-25T18:24:21.207 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:24:21.210 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:24:21.210 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:24:23.799 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:24:23.799 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:25:08.439 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:25:08.528 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:15.542 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:15.817 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:25:15.817 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:25:15.817 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:25:15.817 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:15.818 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:25:15.818 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:25:16.148 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:25:16.723 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:25:32 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:25:16.878 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:25:16.879 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-25T18:25:16.879 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T18:25:16.879 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-25T18:25:16.879 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:25:16.879 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:25:16.879 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:25:16.879 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:25:16.879 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================== 2024-03-25T18:25:16.879 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:25:16.879 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:25:16.880 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.4 M 2024-03-25T18:25:16.880 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 6.4 M 2024-03-25T18:25:16.880 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:25:19.712 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 516 kB/s | 1.4 MB 00:02 2024-03-25T18:25:19.713 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:19.713 INFO:teuthology.orchestra.run.smithi052.stdout:Total 516 kB/s | 1.4 MB 00:02 2024-03-25T18:25:19.713 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:25:19.717 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:25:19.717 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:25:19.764 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:25:19.764 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:25:20.034 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:25:20.129 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:25:20.600 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:25:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:25:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:25:20.868 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:25:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:20.868 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:25:20.869 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:25:21.051 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:25:21.622 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:25:37 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:25:21.777 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:25:21.778 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:25:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T18:25:21.778 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:25:21.778 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:25:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:25:21.778 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:25:21.779 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:25:21.779 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:25:21.779 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:25:21.779 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:25:21.779 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 176 M 2024-03-25T18:25:21.779 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 753 M 2024-03-25T18:25:21.779 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:25:32.062 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 451 kB/s | 416 MB 15:44 2024-03-25T18:25:32.062 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:32.063 INFO:teuthology.orchestra.run.smithi179.stdout:Total 451 kB/s | 416 MB 15:44 2024-03-25T18:25:32.063 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:25:32.067 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:25:32.067 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:25:34.540 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:25:34.540 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:26:16.628 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:26:16.731 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:25.004 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:25.270 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:25.270 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:26:25.270 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:26:25.270 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:25.270 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:26:25.270 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:26:25.498 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:26:26.058 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:26:46 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:26:26.206 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:26:26.207 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-25T18:26:26.207 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T18:26:26.207 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-25T18:26:26.207 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:26:26.207 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:26:26.207 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:26:26.207 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:26:26.207 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-25T18:26:26.208 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T18:26:26.208 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:26:26.208 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 1.4 M 2024-03-25T18:26:26.208 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 6.4 M 2024-03-25T18:26:26.208 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:26:31.120 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 298 kB/s | 1.4 MB 00:04 2024-03-25T18:26:31.120 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:26:31.121 INFO:teuthology.orchestra.run.smithi179.stdout:Total 297 kB/s | 1.4 MB 00:04 2024-03-25T18:26:31.121 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:26:31.125 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:26:31.125 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:26:31.170 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:26:31.170 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:26:31.427 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:26:31.518 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:31.910 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:26:32.181 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:26:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:32.181 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:26:32.181 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:26:32.417 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:26:32.974 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:26:52 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:26:33.121 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:26:33.121 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:26:33.121 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T18:26:33.122 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:26:33.122 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:26:33.122 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:26:33.122 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:26:33.122 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:26:33.122 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:26:33.122 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T18:26:33.122 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:26:33.122 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 176 M 2024-03-25T18:26:33.122 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 753 M 2024-03-25T18:26:33.122 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:29:20.015 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 755 kB/s | 176 MB 03:58 2024-03-25T18:29:20.016 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:29:20.016 INFO:teuthology.orchestra.run.smithi052.stdout:Total 755 kB/s | 176 MB 03:58 2024-03-25T18:29:20.016 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:29:20.019 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:29:20.019 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:29:21.130 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:29:21.130 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:29:40.153 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:29:40.255 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:44.851 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:45.110 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:45.110 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:29:45.110 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:29:45.111 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:29:45.111 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:29:45.111 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:29:45.319 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:29:45.911 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:30:01 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:29:46.068 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 672 M 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 3.1 G 2024-03-25T18:29:46.069 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T18:30:39.436 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 730 kB/s | 176 MB 04:06 2024-03-25T18:30:39.436 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:39.436 INFO:teuthology.orchestra.run.smithi179.stdout:Total 730 kB/s | 176 MB 04:06 2024-03-25T18:30:39.436 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T18:30:39.439 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T18:30:39.439 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T18:30:40.496 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T18:30:40.496 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T18:30:58.703 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T18:30:58.964 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:31:03.761 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:31:04.020 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:31:04.021 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:31:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T18:31:04.021 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:04.021 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:31:04.021 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T18:31:04.219 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:31:04.782 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:31:24 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T18:31:04.930 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T18:31:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:31:04.931 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T18:31:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:31:04.931 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T18:31:04.931 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:31:04.932 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:31:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T18:31:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T18:31:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T18:31:04.932 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T18:31:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 672 M 2024-03-25T18:31:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 3.1 G 2024-03-25T18:31:04.932 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T18:58:12.338 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 403 kB/s | 672 MB 28:26 2024-03-25T18:58:12.339 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:58:12.339 INFO:teuthology.orchestra.run.smithi052.stdout:Total 403 kB/s | 672 MB 28:26 2024-03-25T18:58:12.339 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T18:58:12.342 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T18:58:12.343 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T18:58:16.495 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T18:58:16.495 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T18:59:31.582 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T18:59:31.731 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:59:39.153 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:59:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:59:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:59:39.426 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T18:59:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:59:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:59:39.427 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:59:39.712 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:59:40.285 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:59:56 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:59:40.385 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:59:40.440 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:59:40.441 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-03-25T18:59:40.441 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T18:59:40.502 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:59:41.073 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:59:57 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T18:59:41.228 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T18:59:41.228 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:59:41.228 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T18:59:41.228 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:59:41.229 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T18:59:41.229 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:59:41.229 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:59:41.229 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T18:59:41.229 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T18:59:41.229 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T18:59:41.229 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T18:59:41.229 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 20 M 2024-03-25T18:59:41.229 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 103 M 2024-03-25T18:59:41.229 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:00:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 443 kB/s | 20 MB 00:45 2024-03-25T19:00:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:00:26.721 INFO:teuthology.orchestra.run.smithi052.stdout:Total 443 kB/s | 20 MB 00:45 2024-03-25T19:00:26.722 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:00:26.725 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:00:26.725 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:00:26.881 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:00:26.881 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:00:29.244 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:00:29.354 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:30.300 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:30.602 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:30.602 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:00:30.602 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:00:30.602 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:00:30.602 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:00:30.603 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:00:30.775 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:00:31.348 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:00:47 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:00:31.505 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:00:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:00:31.506 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:00:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:00:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:00:31.506 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:00:31.506 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:00:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:00:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:00:31.506 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:00:31.507 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:00:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 48 M 2024-03-25T19:00:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 173 M 2024-03-25T19:00:31.507 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:02:35.477 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 394 kB/s | 48 MB 02:03 2024-03-25T19:02:35.478 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:02:35.478 INFO:teuthology.orchestra.run.smithi052.stdout:Total 394 kB/s | 48 MB 02:03 2024-03-25T19:02:35.478 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:02:35.481 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:02:35.481 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:02:35.809 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:02:35.809 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:02:40.831 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:02:40.919 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:42.254 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:42.527 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:02:42.527 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:02:42.527 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:02:42.527 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:02:42.527 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:02:42.527 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:02:42.725 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:02:43.299 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:02:59 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:02:43.453 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:02:43.454 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:02:43.454 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:02:43.454 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:02:43.454 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:02:43.454 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:02:43.454 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:02:43.454 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:02:43.454 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:02:43.454 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:02:43.454 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:02:43.454 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 99 M 2024-03-25T19:02:43.455 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 449 M 2024-03-25T19:02:43.455 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:02:44.500 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 362 kB/s | 672 MB 31:39 2024-03-25T19:02:44.500 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:02:44.501 INFO:teuthology.orchestra.run.smithi179.stdout:Total 362 kB/s | 672 MB 31:39 2024-03-25T19:02:44.501 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:02:44.506 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:02:44.506 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:02:48.467 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:02:48.467 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:03:59.283 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:03:59.384 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:04:09.822 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:04:10.093 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:04:10.094 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:04:10.094 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:04:10.094 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:10.094 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:04:10.094 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:04:10.384 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:04:10.951 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:04:30 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:04:11.048 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:04:11.101 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:04:11.102 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-25T19:04:11.102 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:04:11.166 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:04:11.730 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:04:31 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:04:11.877 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:04:11.878 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:04:11.878 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:04:11.878 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:04:11.879 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:04:11.879 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:04:11.879 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:04:11.879 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:04:11.879 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:04:11.879 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:04:11.879 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:04:11.879 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 20 M 2024-03-25T19:04:11.879 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 103 M 2024-03-25T19:04:11.879 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:04:46.234 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 587 kB/s | 20 MB 00:34 2024-03-25T19:04:46.235 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:46.235 INFO:teuthology.orchestra.run.smithi179.stdout:Total 587 kB/s | 20 MB 00:34 2024-03-25T19:04:46.236 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:04:46.239 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:04:46.239 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:04:46.388 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:04:46.388 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:04:48.620 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:04:48.791 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:49.830 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:04:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:04:50.095 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:04:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:50.095 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:04:50.095 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:04:50.310 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:04:50.888 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:05:10 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:04:51.047 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:04:51.047 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:04:51.047 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:04:51.048 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:04:51.048 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:04:51.048 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:04:51.048 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:04:51.048 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:04:51.048 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:04:51.048 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:04:51.048 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:04:51.049 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 48 M 2024-03-25T19:04:51.049 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 173 M 2024-03-25T19:04:51.049 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:06:23.644 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 528 kB/s | 48 MB 01:32 2024-03-25T19:06:23.644 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:06:23.645 INFO:teuthology.orchestra.run.smithi179.stdout:Total 528 kB/s | 48 MB 01:32 2024-03-25T19:06:23.645 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:06:23.648 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:06:23.648 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:06:23.962 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:06:23.963 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:06:28.710 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:06:28.891 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:06:30.258 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:06:30.534 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:06:30.534 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:06:30.534 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:06:30.534 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:06:30.535 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:06:30.535 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:06:30.687 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:06:31.246 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:06:51 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:06:31.393 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:06:31.394 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:06:31.394 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:06:31.394 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:06:31.394 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:06:31.395 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:06:31.395 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:06:31.395 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:06:31.395 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:06:31.395 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:06:31.395 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:06:31.395 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 99 M 2024-03-25T19:06:31.395 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 449 M 2024-03-25T19:06:31.395 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:06:46.252 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 416 kB/s | 99 MB 04:02 2024-03-25T19:06:46.253 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:06:46.253 INFO:teuthology.orchestra.run.smithi052.stdout:Total 416 kB/s | 99 MB 04:02 2024-03-25T19:06:46.253 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:06:46.257 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:06:46.257 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:06:46.896 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:06:46.897 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:06:57.778 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:06:57.886 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:00.832 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:07:01.102 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:07:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:01.102 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:07:01.102 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:07:01.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:07:01.998 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:07:17 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:07:02.158 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:07:02.159 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:07:02.159 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:07:02.159 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:07:02.159 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:07:02.159 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:07:02.159 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:07:02.159 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:07:02.159 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:07:02.159 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:07:02.159 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:07:02.159 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 287 M 2024-03-25T19:07:02.160 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.2 G 2024-03-25T19:07:02.160 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:10:12.828 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 456 kB/s | 99 MB 03:41 2024-03-25T19:10:12.829 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:10:12.829 INFO:teuthology.orchestra.run.smithi179.stdout:Total 456 kB/s | 99 MB 03:41 2024-03-25T19:10:12.829 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:10:12.832 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:10:12.832 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:10:13.442 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:10:13.442 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:10:23.735 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:10:23.842 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:10:26.813 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:10:27.080 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:10:27.081 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:10:27.081 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:10:27.081 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:10:27.081 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:10:27.081 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:10:27.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:10:27.832 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:10:47 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:10:27.982 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:10:27.983 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:10:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:10:27.983 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:10:27.983 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:10:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:10:27.983 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:10:27.984 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:10:27.984 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:10:27.984 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:10:27.984 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:10:27.984 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 287 M 2024-03-25T19:10:27.984 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 1.2 G 2024-03-25T19:10:27.984 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:18:22.905 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 618 kB/s | 287 MB 07:54 2024-03-25T19:18:22.906 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:22.906 INFO:teuthology.orchestra.run.smithi179.stdout:Total 618 kB/s | 287 MB 07:54 2024-03-25T19:18:22.906 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:18:22.909 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:18:22.909 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:18:24.616 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:18:24.616 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:18:53.916 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:18:54.479 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:58.886 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:59.170 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:59.170 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:18:59.171 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:18:59.171 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:59.171 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:18:59.171 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:18:59.364 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:18:59.934 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:19:19 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:19:00.083 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:19:00.084 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:00.084 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:19:00.084 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:00.084 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:19:00.084 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:19:00.084 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:00.084 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:19:00.084 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:00.084 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:19:00.084 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:00.085 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 5.9 M 2024-03-25T19:19:00.085 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 28 M 2024-03-25T19:19:00.085 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:19:07.529 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 817 kB/s | 5.9 MB 00:07 2024-03-25T19:19:07.530 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:07.531 INFO:teuthology.orchestra.run.smithi179.stdout:Total 817 kB/s | 5.9 MB 00:07 2024-03-25T19:19:07.531 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:19:07.535 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:19:07.535 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:19:07.606 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:19:07.607 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:19:08.342 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:19:08.445 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:19:09.002 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:19:09.304 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:19:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:09.305 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:19:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:09.305 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:09.305 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:19:09.509 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:19:10.070 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:19:30 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:19:10.222 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 109 k 2024-03-25T19:19:10.223 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 440 k 2024-03-25T19:19:10.224 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:19:10.810 INFO:teuthology.orchestra.run.smithi179.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 185 kB/s | 109 kB 00:00 2024-03-25T19:19:10.811 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:10.811 INFO:teuthology.orchestra.run.smithi179.stdout:Total 185 kB/s | 109 kB 00:00 2024-03-25T19:19:10.811 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:19:10.814 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:19:10.814 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:19:10.848 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:19:10.848 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:19:10.959 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:19:11.065 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:19:11.456 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:19:11.721 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:19:11.721 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:11.721 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:19:11.721 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:11.721 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:11.721 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:19:11.898 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:19:12.468 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:19:32 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:19:12.618 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:19:12.618 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:12.618 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:19:12.618 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:12.618 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:19:12.618 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:19:12.618 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:12.618 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:19:12.619 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:12.619 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:19:12.619 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:12.619 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 5.4 M 2024-03-25T19:19:12.619 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 25 M 2024-03-25T19:19:12.619 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:19:21.295 INFO:teuthology.orchestra.run.smithi179.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 632 kB/s | 5.4 MB 00:08 2024-03-25T19:19:21.296 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:21.296 INFO:teuthology.orchestra.run.smithi179.stdout:Total 632 kB/s | 5.4 MB 00:08 2024-03-25T19:19:21.297 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:19:21.301 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:19:21.301 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:19:21.369 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:19:21.370 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:19:22.042 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:19:22.144 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:19:22.707 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:19:22.978 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:19:22.978 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:22.978 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:19:22.978 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:22.979 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:22.979 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:19:23.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados2-debuginfo 2024-03-25T19:19:23.730 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:19:43 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:19:23.879 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:19:23.879 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 44 M 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 253 M 2024-03-25T19:19:23.880 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:20:43.348 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 358 kB/s | 287 MB 13:41 2024-03-25T19:20:43.348 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:20:43.349 INFO:teuthology.orchestra.run.smithi052.stdout:Total 358 kB/s | 287 MB 13:41 2024-03-25T19:20:43.349 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:20:43.352 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:20:43.352 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:20:45.145 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:20:45.146 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:21:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 449 kB/s | 44 MB 01:40 2024-03-25T19:21:04.795 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:Total 449 kB/s | 44 MB 01:40 2024-03-25T19:21:04.796 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:21:04.799 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:21:04.799 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:21:05.099 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:21:05.099 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:21:10.160 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:21:10.244 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:21:12.146 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:21:12.411 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:21:12.412 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:21:12.412 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:21:12.412 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:21:12.412 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:21:12.412 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:21:12.623 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:21:13.189 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:21:33 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:21:13.338 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:21:13.339 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:21:13.339 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:21:13.339 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:21:13.339 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:21:13.339 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:21:13.339 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:21:13.339 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:21:13.339 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:21:13.339 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:21:13.339 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:21:13.339 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 52 M 2024-03-25T19:21:13.340 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 308 M 2024-03-25T19:21:13.340 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:21:16.345 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:21:16.453 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:23.831 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:24.134 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:21:24.134 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:24.134 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:21:24.134 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:21:24.134 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:24.134 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:21:24.331 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:21:24.906 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:21:40 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:21:25.059 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:21:25.060 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:25.060 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:21:25.060 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:25.060 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:21:25.060 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:21:25.060 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:25.060 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:21:25.060 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:25.060 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:21:25.060 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:25.061 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 5.9 M 2024-03-25T19:21:25.061 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 28 M 2024-03-25T19:21:25.061 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:21:34.522 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 643 kB/s | 5.9 MB 00:09 2024-03-25T19:21:34.531 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:34.531 INFO:teuthology.orchestra.run.smithi052.stdout:Total 643 kB/s | 5.9 MB 00:09 2024-03-25T19:21:34.531 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:21:34.531 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:21:34.531 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:21:34.603 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:21:34.604 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:21:35.377 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:21:35.481 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:21:36.049 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:21:36.303 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:21:36.303 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:36.303 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:21:36.303 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:21:36.303 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:36.303 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:21:36.502 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:21:37.074 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:21:53 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:21:37.232 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:21:37.232 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:37.233 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:21:37.233 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:37.233 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:21:37.233 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:21:37.233 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:37.233 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:21:37.233 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:37.233 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:21:37.233 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:37.234 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 109 k 2024-03-25T19:21:37.234 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 440 k 2024-03-25T19:21:37.234 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:21:37.841 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 179 kB/s | 109 kB 00:00 2024-03-25T19:21:37.841 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:37.842 INFO:teuthology.orchestra.run.smithi052.stdout:Total 179 kB/s | 109 kB 00:00 2024-03-25T19:21:37.842 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:21:37.845 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:21:37.845 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:21:37.882 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:21:37.882 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:21:38.002 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:21:38.101 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:21:38.515 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:21:38.796 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:21:38.796 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:38.796 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:21:38.796 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:21:38.796 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:38.796 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:21:38.983 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:21:39.558 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:21:55 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:21:39.714 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:21:39.714 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:39.714 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:21:39.714 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:39.714 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:21:39.714 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:21:39.714 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:39.714 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:21:39.714 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:39.714 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:21:39.715 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:39.715 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 5.4 M 2024-03-25T19:21:39.715 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 25 M 2024-03-25T19:21:39.715 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:21:51.060 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 483 kB/s | 5.4 MB 00:11 2024-03-25T19:21:51.060 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:21:51.061 INFO:teuthology.orchestra.run.smithi052.stdout:Total 483 kB/s | 5.4 MB 00:11 2024-03-25T19:21:51.061 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:21:51.065 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:21:51.065 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:21:51.138 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:21:51.138 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:21:51.839 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:21:51.961 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:21:52.477 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:21:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:21:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:52.743 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:21:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:21:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:52.743 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:21:52.940 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2-debuginfo 2024-03-25T19:21:53.512 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:22:09 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:21:53.667 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:21:53.668 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:53.668 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:21:53.668 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:53.668 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:21:53.668 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:21:53.668 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:53.668 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:21:53.668 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:21:53.668 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:21:53.668 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:21:53.669 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 44 M 2024-03-25T19:21:53.669 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 253 M 2024-03-25T19:21:53.669 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:22:20.365 INFO:teuthology.orchestra.run.smithi179.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 792 kB/s | 52 MB 01:07 2024-03-25T19:22:20.365 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:20.365 INFO:teuthology.orchestra.run.smithi179.stdout:Total 792 kB/s | 52 MB 01:07 2024-03-25T19:22:20.366 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:22:20.369 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:22:20.369 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:22:20.714 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:22:20.714 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:22:26.693 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:22:26.854 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:22:28.979 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:22:29.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:22:29.250 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:29.250 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:22:29.250 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:29.250 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:29.250 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:22:29.435 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:22:30.001 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:22:50 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:22:30.152 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:22:30.152 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:30.152 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:22:30.152 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:30.152 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:22:30.153 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:22:30.153 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:30.153 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:22:30.153 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:30.153 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:22:30.153 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:30.153 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 615 k 2024-03-25T19:22:30.153 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 2.1 M 2024-03-25T19:22:30.154 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:22:32.267 INFO:teuthology.orchestra.run.smithi179.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 291 kB/s | 615 kB 00:02 2024-03-25T19:22:32.267 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:32.268 INFO:teuthology.orchestra.run.smithi179.stdout:Total 291 kB/s | 615 kB 00:02 2024-03-25T19:22:32.268 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:22:32.272 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:22:32.272 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:22:32.314 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:22:32.314 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:22:32.485 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:22:32.599 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:32.976 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:33.234 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:33.234 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:33.234 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:22:33.234 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:33.234 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:33.234 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:22:33.427 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:22:33.985 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:22:53 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:22:34.135 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:22:34.136 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:22:34.136 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:34.136 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:22:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:22:34.136 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:34.136 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:22:34.136 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:34.136 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:22:34.137 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:34.137 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 771 k 2024-03-25T19:22:34.137 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 2.7 M 2024-03-25T19:22:34.137 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:22:35.503 INFO:teuthology.orchestra.run.smithi179.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 564 kB/s | 771 kB 00:01 2024-03-25T19:22:35.504 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:35.504 INFO:teuthology.orchestra.run.smithi179.stdout:Total 564 kB/s | 771 kB 00:01 2024-03-25T19:22:35.504 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:22:35.508 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:22:35.508 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:22:35.550 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:22:35.550 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:22:35.731 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:22:35.835 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:22:36.203 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:22:36.474 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:22:36.474 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:36.474 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:22:36.474 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:36.474 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:36.474 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:22:36.684 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:22:37.253 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:22:57 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:22:37.403 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:22:37.403 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:37.404 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:22:37.404 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:37.404 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:22:37.404 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:22:37.404 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:37.404 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:22:37.404 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:37.404 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:22:37.404 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:37.405 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 1.1 M 2024-03-25T19:22:37.405 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 4.3 M 2024-03-25T19:22:37.405 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:22:42.523 INFO:teuthology.orchestra.run.smithi179.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 225 kB/s | 1.1 MB 00:05 2024-03-25T19:22:42.523 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:42.523 INFO:teuthology.orchestra.run.smithi179.stdout:Total 225 kB/s | 1.1 MB 00:05 2024-03-25T19:22:42.524 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:22:42.528 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:22:42.528 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:22:42.569 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:22:42.570 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:22:42.792 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:22:42.897 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:43.304 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:43.556 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:43.556 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:43.556 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:22:43.557 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:43.557 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:43.557 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:22:43.787 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:22:44.347 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:23:04 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:22:44.495 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:22:44.496 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:44.496 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:22:44.496 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:44.496 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:22:44.496 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:22:44.496 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:44.496 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:22:44.496 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:44.496 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:22:44.496 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:44.497 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 337 k 2024-03-25T19:22:44.497 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 1.0 M 2024-03-25T19:22:44.497 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:22:45.618 INFO:teuthology.orchestra.run.smithi179.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 301 kB/s | 337 kB 00:01 2024-03-25T19:22:45.618 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:45.618 INFO:teuthology.orchestra.run.smithi179.stdout:Total 301 kB/s | 337 kB 00:01 2024-03-25T19:22:45.619 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:22:45.623 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:22:45.623 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:22:45.662 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:22:45.662 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:22:45.804 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:22:45.900 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:46.259 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:46.515 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:22:46.515 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:46.515 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:22:46.515 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:46.515 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:46.515 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:22:46.702 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:22:47.284 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:23:07 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:22:47.434 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 509 k 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 2.5 M 2024-03-25T19:22:47.435 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:22:49.421 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 256 kB/s | 509 kB 00:01 2024-03-25T19:22:49.422 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:49.422 INFO:teuthology.orchestra.run.smithi179.stdout:Total 256 kB/s | 509 kB 00:01 2024-03-25T19:22:49.422 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:22:49.426 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:22:49.426 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:22:49.468 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:22:49.468 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:22:49.636 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:22:49.736 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:50.130 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:50.414 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:22:50.414 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:50.414 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:22:50.414 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:50.414 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:50.415 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:22:50.635 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:22:51.203 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:23:11 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:22:51.354 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:22:51.385 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:51.385 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:22:51.385 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:51.386 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:22:51.386 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:22:51.386 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:51.386 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:22:51.386 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:22:51.386 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:22:51.386 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:22:51.386 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 52 M 2024-03-25T19:22:51.386 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 298 M 2024-03-25T19:22:51.386 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:23:07.063 INFO:teuthology.orchestra.run.smithi052.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 618 kB/s | 44 MB 01:13 2024-03-25T19:23:07.064 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:07.064 INFO:teuthology.orchestra.run.smithi052.stdout:Total 618 kB/s | 44 MB 01:13 2024-03-25T19:23:07.065 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:23:07.069 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:23:07.069 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:23:07.383 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:23:07.383 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:23:12.753 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:23:12.865 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:14.659 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:14.942 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:23:14.942 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:23:14.942 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:23:14.942 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:14.942 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:23:14.942 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:23:15.130 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:23:15.705 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:23:31 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:23:15.860 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:23:15.860 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:23:15.860 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:23:15.860 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:23:15.860 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:23:15.860 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:23:15.861 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:23:15.861 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:23:15.861 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:23:15.861 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:23:15.861 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:23:15.861 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 52 M 2024-03-25T19:23:15.861 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 308 M 2024-03-25T19:23:15.861 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:23:46.709 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 953 kB/s | 52 MB 00:55 2024-03-25T19:23:46.710 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:46.710 INFO:teuthology.orchestra.run.smithi179.stdout:Total 953 kB/s | 52 MB 00:55 2024-03-25T19:23:46.710 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:23:46.714 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:23:46.714 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:23:47.049 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:23:47.049 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:23:52.962 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:23:53.283 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:23:55.354 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:23:55.620 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:23:55.620 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:23:55.620 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:23:55.620 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:55.620 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:23:55.620 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:23:55.812 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:23:56.388 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 1:24:16 ago on Mon 25 Mar 2024 05:59:40 PM UTC. 2024-03-25T19:23:56.538 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-25T19:23:56.539 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:23:56.539 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-25T19:23:56.539 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:23:56.539 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-25T19:23:56.539 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:23:56.539 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:23:56.539 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-25T19:23:56.540 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-25T19:23:56.540 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-25T19:23:56.540 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:23:56.540 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 1.3 M 2024-03-25T19:23:56.540 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 5.5 M 2024-03-25T19:23:56.540 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-25T19:24:00.661 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 311 kB/s | 1.3 MB 00:04 2024-03-25T19:24:00.662 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:00.662 INFO:teuthology.orchestra.run.smithi179.stdout:Total 311 kB/s | 1.3 MB 00:04 2024-03-25T19:24:00.662 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-25T19:24:00.666 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-25T19:24:00.666 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-25T19:24:00.711 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-25T19:24:00.712 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-25T19:24:00.972 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-25T19:24:01.093 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:24:01.499 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:24:01.761 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:24:01.762 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:24:01.762 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-25T19:24:01.762 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:01.762 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:24:01.762 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-25T19:24:01.969 DEBUG:teuthology.parallel:result is None 2024-03-25T19:24:55.472 INFO:teuthology.orchestra.run.smithi052.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 533 kB/s | 52 MB 01:39 2024-03-25T19:24:55.473 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:55.473 INFO:teuthology.orchestra.run.smithi052.stdout:Total 533 kB/s | 52 MB 01:39 2024-03-25T19:24:55.473 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:24:55.477 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:24:55.477 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:24:55.838 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:24:55.839 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:25:02.189 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:25:02.298 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:04.446 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:04.718 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:25:04.718 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:04.718 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:25:04.718 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:04.718 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:04.719 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:25:04.896 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:25:05.466 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:25:21 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:25:05.621 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:25:05.622 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:05.622 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:25:05.622 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:05.622 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:25:05.622 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:25:05.622 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:05.622 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:25:05.622 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:05.623 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:25:05.623 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:05.623 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 615 k 2024-03-25T19:25:05.623 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.1 M 2024-03-25T19:25:05.623 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:25:06.633 INFO:teuthology.orchestra.run.smithi052.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 609 kB/s | 615 kB 00:01 2024-03-25T19:25:06.634 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:06.634 INFO:teuthology.orchestra.run.smithi052.stdout:Total 608 kB/s | 615 kB 00:01 2024-03-25T19:25:06.635 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:25:06.638 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:25:06.639 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:25:06.682 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:25:06.682 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:25:06.855 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:25:06.960 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:25:07.379 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:25:07.644 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:25:07.644 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:07.644 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:25:07.644 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:07.644 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:07.644 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:25:07.852 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:25:08.427 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:25:24 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:25:08.585 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:25:08.586 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:25:08.586 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:08.586 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:25:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:25:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:08.586 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:25:08.586 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:08.586 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:25:08.586 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:08.587 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 771 k 2024-03-25T19:25:08.587 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.7 M 2024-03-25T19:25:08.587 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:25:10.011 INFO:teuthology.orchestra.run.smithi052.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 541 kB/s | 771 kB 00:01 2024-03-25T19:25:10.012 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:10.012 INFO:teuthology.orchestra.run.smithi052.stdout:Total 541 kB/s | 771 kB 00:01 2024-03-25T19:25:10.012 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:25:10.016 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:25:10.016 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:25:10.060 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:25:10.061 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:25:10.270 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:25:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:25:10.760 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:25:11.026 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:25:11.026 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:11.027 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:25:11.027 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:11.027 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:11.027 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:25:11.255 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:25:11.827 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:25:27 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:25:11.984 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:25:11.985 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:25:11.985 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:11.985 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:25:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:25:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:11.985 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:25:11.985 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:11.985 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:25:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:11.986 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.1 M 2024-03-25T19:25:11.986 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 4.3 M 2024-03-25T19:25:11.986 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:25:15.189 INFO:teuthology.orchestra.run.smithi052.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 359 kB/s | 1.1 MB 00:03 2024-03-25T19:25:15.190 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:15.190 INFO:teuthology.orchestra.run.smithi052.stdout:Total 359 kB/s | 1.1 MB 00:03 2024-03-25T19:25:15.190 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:25:15.194 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:25:15.194 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:25:15.240 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:25:15.240 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:25:15.469 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:25:15.590 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:16.009 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:16.274 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:16.275 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:16.275 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:25:16.275 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:16.275 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:16.275 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:25:16.462 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:25:17.036 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:25:33 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:25:17.192 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 337 k 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.0 M 2024-03-25T19:25:17.193 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:25:18.484 INFO:teuthology.orchestra.run.smithi052.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 261 kB/s | 337 kB 00:01 2024-03-25T19:25:18.484 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:18.485 INFO:teuthology.orchestra.run.smithi052.stdout:Total 261 kB/s | 337 kB 00:01 2024-03-25T19:25:18.485 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:25:18.489 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:25:18.489 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:25:18.530 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:25:18.531 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:25:18.675 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:25:18.777 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:19.153 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:19.408 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:25:19.408 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:19.408 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:25:19.408 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:19.408 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:19.408 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:25:19.614 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:25:20.187 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:25:36 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:25:20.343 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:25:20.343 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 509 k 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 2.5 M 2024-03-25T19:25:20.344 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:25:22.137 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 284 kB/s | 509 kB 00:01 2024-03-25T19:25:22.137 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:22.138 INFO:teuthology.orchestra.run.smithi052.stdout:Total 284 kB/s | 509 kB 00:01 2024-03-25T19:25:22.138 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:25:22.142 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:25:22.142 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:25:22.185 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:25:22.186 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:25:22.358 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:25:22.471 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:22.866 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:23.143 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:25:23.143 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:23.143 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:25:23.143 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:23.143 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:23.143 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:25:23.318 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:25:23.889 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:25:39 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:25:24.044 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 52 M 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 298 M 2024-03-25T19:25:24.045 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:26:18.695 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 966 kB/s | 52 MB 00:54 2024-03-25T19:26:18.695 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:18.696 INFO:teuthology.orchestra.run.smithi052.stdout:Total 966 kB/s | 52 MB 00:54 2024-03-25T19:26:18.696 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:26:18.700 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:26:18.700 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:26:19.051 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:26:19.051 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:26:25.306 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:26:25.404 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:26:27.445 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:26:27.757 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:26:27.758 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:27.758 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:26:27.758 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:27.758 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:27.758 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:26:27.960 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:26:28.531 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 1:26:44 ago on Mon 25 Mar 2024 05:59:44 PM UTC. 2024-03-25T19:26:28.686 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-03-25T19:26:28.687 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:26:28.687 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-03-25T19:26:28.687 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:26:28.687 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-03-25T19:26:28.687 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:26:28.687 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:28.687 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-03-25T19:26:28.687 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-03-25T19:26:28.687 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-03-25T19:26:28.687 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:28.687 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.3 M 2024-03-25T19:26:28.688 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 5.5 M 2024-03-25T19:26:28.688 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-03-25T19:26:30.738 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 625 kB/s | 1.3 MB 00:02 2024-03-25T19:26:30.739 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:30.739 INFO:teuthology.orchestra.run.smithi052.stdout:Total 625 kB/s | 1.3 MB 00:02 2024-03-25T19:26:30.740 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-03-25T19:26:30.744 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-03-25T19:26:30.744 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-03-25T19:26:30.791 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-03-25T19:26:30.792 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-03-25T19:26:31.039 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-03-25T19:26:31.140 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:26:31.684 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:26:31.959 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:26:31.959 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:31.959 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-03-25T19:26:31.959 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:31.959 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:31.959 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-03-25T19:26:32.255 DEBUG:teuthology.parallel:result is None 2024-03-25T19:26:32.256 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:26:32.398 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:26:32.418 INFO:teuthology.orchestra.run.smithi052.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:26:32.418 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:26:32.418 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:26:32.420 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:26:32.630 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:26:32.654 INFO:teuthology.orchestra.run.smithi179.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:26:32.655 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:26:32.655 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:26:32.656 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:26:32.656 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:26:32.656 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:26:32.683 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:26:32.683 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:26:32.726 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:26:32.726 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:26:32.726 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:26:32.752 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:26:32.819 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:26:32.819 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:26:32.851 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:26:32.920 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:26:32.920 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:26:32.920 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:26:32.947 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:26:33.013 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:26:33.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:26:33.043 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:26:33.113 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:26:33.113 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:26:33.113 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:26:33.139 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:26:33.205 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:26:33.206 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:26:33.235 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:26:33.304 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:26:33.387 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:26:33.387 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /var/log/ceph 2024-03-25T19:26:33.389 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /var/log/ceph 2024-03-25T19:26:33.417 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:26:33.418 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:26:33.456 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:26:33.486 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:26:33.486 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:26:33.523 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:26:33.564 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:26:33.564 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:26:33.564 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621315', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi052.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFu75LBK5+mmCXqXFQkEaeqx9olToVK74M8E1goFv+hIIYcWKlNBJAmGrZuaHX/fnZlzu8o4Ef/fcApmHnY2UWA=', 'smithi179.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPoGCLeTMdvJXt4cXBSaRu+M6PrJ8QIjxvJD4/CWoPQiXProONOev5+Tv0UkCP3AYZH/yTtc4C7WCDgX2lCfAyY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:26:33.565 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:26:33.589 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:26:33.610 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:26:33.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:26:33.677 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:26:33.677 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:26:33.722 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:26:33.722 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2024-03-25T19:26:33.776 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:26:33.777 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:26:33.777 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:26:33.777 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:26:33.777 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:26:33.777 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-25 18:05:58.947652903 +0000 2024-03-25T19:26:33.777 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-25 17:58:35.527583882 +0000 2024-03-25T19:26:33.777 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-25 17:58:35.527583882 +0000 2024-03-25T19:26:33.777 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-25 17:58:35.527583882 +0000 2024-03-25T19:26:33.777 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:26:33.842 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-25T19:26:33.842 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-25T19:26:33.842 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.6844e-05 s, 5.3 MB/s 2024-03-25T19:26:33.844 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:26:33.899 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2024-03-25T19:26:33.953 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:26:33.953 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:26:33.953 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T19:26:33.953 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:26:33.954 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:26:33.954 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-25 18:05:58.948652889 +0000 2024-03-25T19:26:33.954 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-25 17:58:35.852579692 +0000 2024-03-25T19:26:33.954 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-25 17:58:35.852579692 +0000 2024-03-25T19:26:33.954 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-25 17:58:35.852579692 +0000 2024-03-25T19:26:33.954 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:26:34.018 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-25T19:26:34.018 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-25T19:26:34.018 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.3777e-05 s, 5.5 MB/s 2024-03-25T19:26:34.022 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:26:34.077 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2024-03-25T19:26:34.131 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:26:34.131 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:26:34.132 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T19:26:34.132 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:26:34.132 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:26:34.132 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-25 18:05:58.948652889 +0000 2024-03-25T19:26:34.132 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-25 17:58:36.188575361 +0000 2024-03-25T19:26:34.132 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-25 17:58:36.188575361 +0000 2024-03-25T19:26:34.132 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-25 17:58:36.188575361 +0000 2024-03-25T19:26:34.132 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:26:34.196 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-25T19:26:34.196 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-25T19:26:34.196 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.4338e-05 s, 5.4 MB/s 2024-03-25T19:26:34.197 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:26:34.252 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2024-03-25T19:26:34.306 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:26:34.306 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:26:34.306 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T19:26:34.306 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:26:34.307 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:26:34.307 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-03-25 18:05:58.949652877 +0000 2024-03-25T19:26:34.307 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-03-25 17:58:36.526571004 +0000 2024-03-25T19:26:34.307 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-03-25 17:58:36.526571004 +0000 2024-03-25T19:26:34.307 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-03-25 17:58:36.526571004 +0000 2024-03-25T19:26:34.307 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:26:34.371 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-03-25T19:26:34.371 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-03-25T19:26:34.371 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.4732e-05 s, 5.4 MB/s 2024-03-25T19:26:34.372 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:26:34.427 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:26:34.427 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:26:34.427 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:26:34.446 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:26:34.446 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2024-03-25T19:26:34.503 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:26:34.504 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:26:34.504 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:26:34.504 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:26:34.504 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:26:34.504 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-25 18:06:33.822296349 +0000 2024-03-25T19:26:34.504 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-25 17:58:35.452634785 +0000 2024-03-25T19:26:34.504 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-25 17:58:35.452634785 +0000 2024-03-25T19:26:34.504 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-25 17:58:35.452634785 +0000 2024-03-25T19:26:34.504 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:26:34.572 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-25T19:26:34.572 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-25T19:26:34.572 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000216167 s, 2.4 MB/s 2024-03-25T19:26:34.574 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:26:34.631 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2024-03-25T19:26:34.688 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:26:34.688 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:26:34.688 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T19:26:34.688 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:26:34.688 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:26:34.688 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-25 18:06:33.822296349 +0000 2024-03-25T19:26:34.688 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-25 17:58:35.798627388 +0000 2024-03-25T19:26:34.688 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-25 17:58:35.798627388 +0000 2024-03-25T19:26:34.688 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-25 17:58:35.798627388 +0000 2024-03-25T19:26:34.689 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:26:34.755 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-25T19:26:34.755 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-25T19:26:34.755 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000237601 s, 2.2 MB/s 2024-03-25T19:26:34.756 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:26:34.813 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2024-03-25T19:26:34.868 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:26:34.869 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:26:34.869 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T19:26:34.869 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:26:34.869 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:26:34.869 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-25 18:06:33.822296349 +0000 2024-03-25T19:26:34.869 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-25 17:58:36.138620119 +0000 2024-03-25T19:26:34.869 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-25 17:58:36.138620119 +0000 2024-03-25T19:26:34.869 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-25 17:58:36.138620119 +0000 2024-03-25T19:26:34.869 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:26:34.935 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-25T19:26:34.935 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-25T19:26:34.935 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000210685 s, 2.4 MB/s 2024-03-25T19:26:34.936 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:26:34.992 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2024-03-25T19:26:35.048 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:26:35.048 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:26:35.048 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:26:35.048 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:26:35.048 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:26:35.048 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-25 18:06:33.823296328 +0000 2024-03-25T19:26:35.048 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-25 17:58:36.482612764 +0000 2024-03-25T19:26:35.048 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-25 17:58:36.482612764 +0000 2024-03-25T19:26:35.048 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-25 17:58:36.482612764 +0000 2024-03-25T19:26:35.048 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:26:35.113 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-25T19:26:35.113 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-25T19:26:35.114 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000178326 s, 2.9 MB/s 2024-03-25T19:26:35.115 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:26:35.170 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:26:35.170 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:26:35.170 INFO:tasks.ceph:Generating config... 2024-03-25T19:26:35.171 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:26:35.171 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:26:35.171 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:26:35.171 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:26:35.171 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:26:35.171 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:26:35.171 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:26:35.171 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:26:35.171 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:26:35.171 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:26:35.171 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:26:35.172 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:26:35.172 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:26:35.172 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:26:35.172 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:26:35.172 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:26:35.172 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:26:35.172 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:26:35.173 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:26:35.174 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:26:35.174 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:26:35.174 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:26:35.174 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:26:35.174 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:26:35.174 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:26:35.174 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:26:35.174 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:26:35.174 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:26:35.174 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:26:35.234 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:26:35.237 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:26:35.283 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:26:35.309 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.52:6789]'), ('mon.c', '[v1:172.21.15.52:6790]'), ('mon.b', '[v1:172.21.15.179:6789]')] 2024-03-25T19:26:35.310 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.52:6789],[v1:172.21.15.52:6790],[v1:172.21.15.179:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:26:35.310 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:26:35.310 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:26:35.364 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.52:6789]' --addv c '[v1:172.21.15.52:6790]' --addv b '[v1:172.21.15.179:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:26:35.449 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:26:35.449 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: generated fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:26:35.449 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = quincy 2024-03-25T19:26:35.449 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 0 2024-03-25T19:26:35.449 INFO:teuthology.orchestra.run.smithi052.stdout:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:26:35.449 INFO:teuthology.orchestra.run.smithi052.stdout:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:26:35.449 INFO:teuthology.orchestra.run.smithi052.stdout:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:26:35.449 INFO:teuthology.orchestra.run.smithi052.stdout:min_mon_release 17 (quincy) 2024-03-25T19:26:35.449 INFO:teuthology.orchestra.run.smithi052.stdout:election_strategy: 3 2024-03-25T19:26:35.449 INFO:teuthology.orchestra.run.smithi052.stdout:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:26:35.449 INFO:teuthology.orchestra.run.smithi052.stdout:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:26:35.450 INFO:teuthology.orchestra.run.smithi052.stdout:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:26:35.450 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:26:35.450 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:26:35.464 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ef6405ee-982f-417b-9446-354282e3761f... 2024-03-25T19:26:35.465 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:26:35.507 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2024-03-25T19:26:35.554 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:26:35.555 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2024-03-25T19:26:35.556 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = "[v1:172.21.15.52:6789],[v1:172.21.15.52:6790],[v1:172.21.15.179:6789]" 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: mon client directed command retry = 5 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: mon election default strategy = 3 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: ms bind msgr2 = False 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: ms inject socket failures = 5000 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: ms type = async 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: osd heartbeat grace = 80 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2024-03-25T19:26:35.557 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug trim objects = true 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:26:35.558 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: bdev async discard = True 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: bdev enable discard = True 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore allocator = stupid 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore block size = 96636764160 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore fsck on mount = True 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: bluestore zero block detection = True 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluefs = 1 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluestore = 1 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: debug monc = 20 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 20 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: debug rocksdb = 10 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:26:35.559 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd full ratio = 0.9 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd class default list = * 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd class load list = * 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd client watch timeout = 120 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify cached snaps = True 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify missing on start = True 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd fast shutdown = False 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd max markdown count = 1000 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd mclock override recovery settings = True 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd objectstore = bluestore 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd op queue = debug_random 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd op queue cut off = debug_random 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: osd shutdown pgref assert = True 2024-03-25T19:26:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2024-03-25T19:26:35.561 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: # 1m isn't quite enough 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:26:35.562 INFO:teuthology.orchestra.run.smithi052.stdout: mon min osdmap epochs = 50 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd crush smoke test = False 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune interval = 2 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune min = 15 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: mon scrub interval = 300 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on pool no app = False 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: paxos service trim min = 10 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: debug monc = 20 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: debug objecter = 20 2024-03-25T19:26:35.563 INFO:teuthology.orchestra.run.smithi052.stdout: debug rados = 20 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.c] 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.b] 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi179.stdout:[global] 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi179.stdout: chdir = "" 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi179.stdout: auth supported = cephx 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi179.stdout: filestore xattr use omap = true 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi179.stdout: mon clock drift allowed = 1.000 2024-03-25T19:26:35.564 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: osd crush chooseleaf type = 0 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: auth debug = true 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on old message = true 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on bug = true 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: mon pg warn max object skew = 0 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default size = 2 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.565 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow primary affinity = true 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow pg remap = true 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on no sortbitwise = false 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on too few osds = false 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon_allow_pool_size_one = true 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: osd default data pool replay window = 5 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: mon allow pool delete = true 2024-03-25T19:26:35.566 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: mon cluster log file level = debug 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: debug asserts on shutdown = true 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: mon health detail to clog = false 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: mon host = "[v1:172.21.15.52:6789],[v1:172.21.15.52:6790],[v1:172.21.15.179:6789]" 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: mon client directed command retry = 5 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: mon election default strategy = 3 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: ms bind msgr2 = False 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: ms inject socket failures = 5000 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: ms type = async 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: osd heartbeat grace = 80 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: fsid = ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout:[osd] 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: osd journal size = 100 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub load threshold = 5.0 2024-03-25T19:26:35.567 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub max interval = 600 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: osd recover clone overlap = true 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: osd recovery max chunk = 1048576 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug shutdown = true 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug op order = true 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify stray on activate = true 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug trim objects = true 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: osd open classes on start = true 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug pg log writeout = true 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.568 INFO:teuthology.orchestra.run.smithi179.stdout: osd map max advance = 10 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: journal zero on create = true 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: filestore apply finisher threads = 3 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: bdev debug aio = true 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug misdirected ops = true 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: bdev async discard = True 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: bdev enable discard = True 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore allocator = stupid 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore block size = 96636764160 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore fsck on mount = True 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore zero block detection = True 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluefs = 1 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluestore = 1 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: debug monc = 20 2024-03-25T19:26:35.569 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: debug osd = 20 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: debug rocksdb = 10 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd full ratio = 0.9 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: osd class default list = * 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: osd class load list = * 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: osd client watch timeout = 120 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify cached snaps = True 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify missing on start = True 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: osd fast shutdown = False 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: osd max markdown count = 1000 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: osd mclock override recovery settings = True 2024-03-25T19:26:35.570 INFO:teuthology.orchestra.run.smithi179.stdout: osd objectstore = bluestore 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: osd op queue = debug_random 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: osd op queue cut off = debug_random 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: osd shutdown pgref assert = True 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr] 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: debug mgr = 20 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: mgr/telemetry/nag = false 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout:[mon] 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: debug paxos = 20 2024-03-25T19:26:35.571 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: mon data avail warn = 5 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: mon mgr mkfs grace = 240 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd reporter subtree level = osd 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd prime pg temp = true 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: # 1m isn't quite enough 2024-03-25T19:26:35.572 INFO:teuthology.orchestra.run.smithi179.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: mon min osdmap epochs = 50 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd crush smoke test = False 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune interval = 2 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune min = 15 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: mon scrub interval = 300 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on pool no app = False 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: paxos service trim min = 10 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout:[client] 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: rgw cache enabled = true 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable ops log = true 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable usage log = true 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:26:35.573 INFO:teuthology.orchestra.run.smithi179.stdout: debug monc = 20 2024-03-25T19:26:35.574 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-25T19:26:35.574 INFO:teuthology.orchestra.run.smithi179.stdout: debug objecter = 20 2024-03-25T19:26:35.574 INFO:teuthology.orchestra.run.smithi179.stdout: debug rados = 20 2024-03-25T19:26:35.574 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.a] 2024-03-25T19:26:35.574 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.c] 2024-03-25T19:26:35.574 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.b] 2024-03-25T19:26:35.575 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:26:35.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:26:35.629 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:26:35.629 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:26:35.629 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:26:35.643 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:26:35.644 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:26:35.697 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2024-03-25T19:26:35.697 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:26:35.697 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:26:35.697 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:26:35.776 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:26:35.776 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:26:35.831 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi179.front.sepia.ceph.com 2024-03-25T19:26:35.831 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:26:35.831 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:26:35.831 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:26:35.871 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:26:35.871 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:26:35.926 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:26:35.926 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:26:35.926 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:26:35.979 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:26:35.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:26:36.046 INFO:teuthology.orchestra.run.smithi179.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:26:36.048 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:26:36.049 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:26:36.049 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:26:36.090 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:26:36.105 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:26:36.147 INFO:teuthology.orchestra.run.smithi179.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:26:36.163 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:26:36.163 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:26:36.163 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:26:36.188 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:26:36.188 INFO:tasks.ceph:role: osd.0 2024-03-25T19:26:36.189 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-25T19:26:36.189 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:26:36.254 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:26:36.254 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:26:36.254 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:26:36.254 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:26:36.254 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:26:36.254 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:26:36.254 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:26:36.254 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:26:36.254 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:26:36.254 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:26:36.286 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-25T19:26:36.288 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-25T19:26:36.288 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:26:36.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:26:36.430 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:26:36.496 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:26:36.497 INFO:tasks.ceph:role: osd.1 2024-03-25T19:26:36.497 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-25T19:26:36.497 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:26:36.563 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:26:36.563 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:26:36.564 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:26:36.564 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:26:36.564 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:26:36.564 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:26:36.564 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:26:36.564 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:26:36.564 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:26:36.564 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:26:36.595 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-25T19:26:36.597 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-25T19:26:36.597 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:26:36.667 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:26:36.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:26:36.804 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:26:36.804 INFO:tasks.ceph:role: osd.2 2024-03-25T19:26:36.805 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-25T19:26:36.805 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:26:36.871 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:26:36.871 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:26:36.871 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:26:36.871 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:26:36.871 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:26:36.871 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:26:36.871 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:26:36.872 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:26:36.872 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:26:36.872 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:26:36.902 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-25T19:26:36.904 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-25T19:26:36.905 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:26:36.985 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:26:37.055 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:26:37.123 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:26:37.123 INFO:tasks.ceph:role: osd.3 2024-03-25T19:26:37.123 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-25T19:26:37.123 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:26:37.189 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:26:37.189 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:26:37.189 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:26:37.189 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:26:37.189 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:26:37.189 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:26:37.190 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:26:37.190 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:26:37.190 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:26:37.190 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:26:37.222 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-03-25T19:26:37.223 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-03-25T19:26:37.223 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:26:37.293 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:26:37.363 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:26:37.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:37.459+0000 7f98044a2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:26:37.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:37.459+0000 7f98044a2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:26:37.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:37.459+0000 7f98044a2640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:26:37.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:37.459+0000 7f98044a2640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:26:37.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:37.459+0000 7f98044a2640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:26:37.459 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:37.460+0000 7f98044a2640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:26:40.300 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:26:40.330 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:26:40.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:40.420+0000 7f8164ed9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:26:40.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:40.420+0000 7f8164ed9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:26:40.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:40.420+0000 7f8164ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:26:40.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:40.420+0000 7f8164ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:26:40.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:40.420+0000 7f8164ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:26:40.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:40.420+0000 7f8164ed9640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:26:43.271 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:26:43.301 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:26:43.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:43.393+0000 7ffbe6ab3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:26:43.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:43.393+0000 7ffbe6ab3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:26:43.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:43.393+0000 7ffbe6ab3640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:26:43.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:43.393+0000 7ffbe6ab3640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:26:43.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:43.393+0000 7ffbe6ab3640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:26:43.393 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:43.394+0000 7ffbe6ab3640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:26:46.236 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:26:46.267 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:26:46.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:46.359+0000 7fcc06dbb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:26:46.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:46.360+0000 7fcc06dbb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:26:46.359 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:46.360+0000 7fcc06dbb640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:26:46.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:46.360+0000 7fcc06dbb640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:26:46.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:46.360+0000 7fcc06dbb640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:26:46.360 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:26:46.360+0000 7fcc06dbb640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:26:49.205 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:26:49.235 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:26:49.267 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:26:49.267 INFO:tasks.ceph:role: osd.4 2024-03-25T19:26:49.267 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-25T19:26:49.267 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:26:49.345 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:26:49.346 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:26:49.346 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:26:49.346 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:26:49.346 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:26:49.346 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:26:49.346 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:26:49.346 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:26:49.346 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:26:49.346 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:26:49.851 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-25T19:26:49.853 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-25T19:26:49.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:26:50.003 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:26:50.042 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:26:50.112 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:26:50.112 INFO:tasks.ceph:role: osd.5 2024-03-25T19:26:50.112 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-25T19:26:50.112 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:26:50.188 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:26:50.188 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:26:50.188 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:26:50.188 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:26:50.188 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:26:50.188 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:26:50.189 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:26:50.189 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:26:50.189 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:26:50.189 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:26:50.640 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-25T19:26:50.643 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-25T19:26:50.643 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:26:50.743 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:26:50.777 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:26:50.846 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:26:50.846 INFO:tasks.ceph:role: osd.6 2024-03-25T19:26:50.846 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-25T19:26:50.846 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:26:50.922 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:26:50.922 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:26:50.922 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:26:50.922 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:26:50.922 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:26:50.922 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:26:50.922 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:26:50.922 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:26:50.923 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:26:50.923 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:26:51.412 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-25T19:26:51.415 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-25T19:26:51.415 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:26:51.568 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:26:51.601 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:26:51.670 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:26:51.670 INFO:tasks.ceph:role: osd.7 2024-03-25T19:26:51.670 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-25T19:26:51.671 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:26:51.746 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:26:51.746 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:26:51.747 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:26:51.747 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:26:51.747 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:26:51.747 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:26:51.747 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:26:51.747 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:26:51.747 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:26:51.747 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:26:52.258 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-25T19:26:52.260 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-25T19:26:52.260 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:26:52.393 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:26:52.429 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:26:52.539 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:52.539+0000 7efcdce66640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:26:52.540 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:52.540+0000 7efcdce66640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:26:52.540 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:52.540+0000 7efcdce66640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:26:52.540 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:52.540+0000 7efcdce66640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:26:52.540 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:52.540+0000 7efcdce66640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:26:52.540 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:52.540+0000 7efcdce66640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:26:55.385 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:26:55.417 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:26:55.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:55.507+0000 7fd27380d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:26:55.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:55.508+0000 7fd27380d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:26:55.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:55.508+0000 7fd27380d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:26:55.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:55.508+0000 7fd27380d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:26:55.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:55.508+0000 7fd27380d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:26:55.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:55.508+0000 7fd27380d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:26:58.351 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:26:58.381 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:26:58.472 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:58.472+0000 7f9627ed7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:26:58.472 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:58.472+0000 7f9627ed7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:26:58.472 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:58.472+0000 7f9627ed7640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:26:58.472 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:58.472+0000 7f9627ed7640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:26:58.472 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:58.472+0000 7f9627ed7640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:26:58.473 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:26:58.473+0000 7f9627ed7640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:27:01.315 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:27:01.346 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:01.437 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:01.436+0000 7f32bc2c8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:27:01.437 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:01.436+0000 7f32bc2c8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:27:01.437 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:01.436+0000 7f32bc2c8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:27:01.437 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:01.436+0000 7f32bc2c8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:27:01.437 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:01.436+0000 7f32bc2c8640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:27:01.437 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:01.436+0000 7f32bc2c8640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:27:04.281 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:27:04.312 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:27:04.313 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:27:04.313 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:27:04.344 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:27:04.344 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:27:04.415 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:27:04.415 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:27:04.484 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:27:04.484 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:27:04.553 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:27:04.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:27:04.622 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:27:04.622 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:27:04.654 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:27:04.654 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:27:04.724 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:27:04.724 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:27:04.794 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:27:04.794 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:27:04.863 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:27:04.863 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:27:04.931 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:27:04.932 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:27:04.948 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:27:04.948 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:27:04.987 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:27:04.987 DEBUG:teuthology.orchestra.run.smithi052:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:27:04.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:27:05.031 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.y] 2024-03-25T19:27:05.031 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDrzwFmYfh3OhAAygL2sMH7egfiMTck60/dhg== 2024-03-25T19:27:05.031 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2024-03-25T19:27:05.031 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDtzwFmB8RrGxAAoIImd8Wo4TnOFLSQdd/38Q== 2024-03-25T19:27:05.031 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2024-03-25T19:27:05.031 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDwzwFmD80TGRAAR6pLQyDot8uDWAGznNLePg== 2024-03-25T19:27:05.031 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDzzwFmII+AFxAAYmGEyM4loVSg8Sf5s8U9vw== 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.3] 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD2zwFmDPZ8FRAA23ZfWbmbkrxkhdG4+OVwWw== 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.x] 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDszwFmalHTAhAAvjIkwAr0vmhWm6jkjA7dNQ== 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.4] 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD8zwFmhmE6IBAA4XJa7nsBVyC/8NXfkt/HGA== 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.5] 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQD/zwFm/P5THhAAmLMmZrAawoQ7Fllbssuezw== 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.6] 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQAC0AFmgI41HBAAdwg3WDPtF56b/32MJiK8tg== 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.7] 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQAF0AFmGgwaGhAAhQL03srjx1psKMxZUgziIA== 2024-03-25T19:27:05.032 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2024-03-25T19:27:05.033 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDszwFmAsp7BRAAOUZ01M6lRDH1vc+iVmEbjw== 2024-03-25T19:27:05.033 INFO:teuthology.orchestra.run.smithi052.stdout:[client.1] 2024-03-25T19:27:05.033 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDszwFmJEngCBAAIS0y7YRO5apuCLY7GdQUKQ== 2024-03-25T19:27:05.054 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr.y] 2024-03-25T19:27:05.054 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQDrzwFmYfh3OhAAygL2sMH7egfiMTck60/dhg== 2024-03-25T19:27:05.054 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.0] 2024-03-25T19:27:05.054 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQDtzwFmB8RrGxAAoIImd8Wo4TnOFLSQdd/38Q== 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.1] 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQDwzwFmD80TGRAAR6pLQyDot8uDWAGznNLePg== 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.2] 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQDzzwFmII+AFxAAYmGEyM4loVSg8Sf5s8U9vw== 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.3] 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQD2zwFmDPZ8FRAA23ZfWbmbkrxkhdG4+OVwWw== 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr.x] 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQDszwFmalHTAhAAvjIkwAr0vmhWm6jkjA7dNQ== 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.4] 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQD8zwFmhmE6IBAA4XJa7nsBVyC/8NXfkt/HGA== 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.5] 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQD/zwFm/P5THhAAmLMmZrAawoQ7Fllbssuezw== 2024-03-25T19:27:05.055 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.6] 2024-03-25T19:27:05.056 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQAC0AFmgI41HBAAdwg3WDPtF56b/32MJiK8tg== 2024-03-25T19:27:05.056 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.7] 2024-03-25T19:27:05.056 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQAF0AFmGgwaGhAAhQL03srjx1psKMxZUgziIA== 2024-03-25T19:27:05.056 INFO:teuthology.orchestra.run.smithi179.stdout:[client.0] 2024-03-25T19:27:05.056 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQDszwFmAsp7BRAAOUZ01M6lRDH1vc+iVmEbjw== 2024-03-25T19:27:05.056 INFO:teuthology.orchestra.run.smithi179.stdout:[client.1] 2024-03-25T19:27:05.056 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQDszwFmJEngCBAAIS0y7YRO5apuCLY7GdQUKQ== 2024-03-25T19:27:05.057 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:27:05.079 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:27:05.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.154 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.207 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.209 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.260 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.262 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.313 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.315 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.368 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:27:05.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:27:05.422 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.424 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.476 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.478 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.529 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.532 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.583 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.585 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:27:05.637 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:27:05.639 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:27:05.694 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:27:05.696 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:27:05.751 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:27:05.751 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:27:05.777 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:27:05.873 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:27:05.899 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:27:05.966 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:27:06.062 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:27:06.088 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:27:06.114 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:27:06.212 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:27:06.239 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:06.241 DEBUG:teuthology.orchestra.run.smithi179:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:27:06.293 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:27:06.293 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:27:06.294 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:27:06.294 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:27:06.296 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:27:06.296 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:27:06.296 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:27:06.296 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:27:06.298 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:27:06.298 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:27:06.298 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:27:06.298 DEBUG:teuthology.orchestra.run.smithi179:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:27:06.336 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:27:06.336 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:27:06.337 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:27:06.337 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:27:06.339 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:27:06.339 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:27:06.339 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:27:06.341 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:27:06.342 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:27:06.342 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:27:06.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.442+0000 7f269aad9640 10 monclient: get_monmap_and_config 2024-03-25T19:27:06.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.442+0000 7f269aad9640 10 monclient: build_initial_monmap 2024-03-25T19:27:06.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.442+0000 7f269aad9640 10 monclient: monmap: 2024-03-25T19:27:06.443 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-25T19:27:06.443 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:06.443 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:27:06.444442+0000 2024-03-25T19:27:06.443 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:27:06.444442+0000 2024-03-25T19:27:06.443 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:06.444 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:06.444 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:06.444 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:06.444 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:06.444 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:06.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.443+0000 7f269aad9640 1 Processor -- start 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.443+0000 7f269aad9640 1 -- start start 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.443+0000 7f269aad9640 10 monclient: init 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f269aad9640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f269aad9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f269aad9640 10 monclient(hunting): picked mon.noname-a con 0x7f26940b4e90 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f269aad9640 10 monclient(hunting): picked mon.noname-b con 0x7f26940b4340 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f269aad9640 10 monclient(hunting): picked mon.noname-c con 0x7f269413fd00 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f269aad9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940b5270 con 0x7f26940b4e90 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f269aad9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940ab330 con 0x7f269413fd00 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f269aad9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940a9d80 con 0x7f26940b4340 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f269aad9640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f269aad9640 10 monclient(hunting): authenticate will time out at 5874.625206s 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f2693fff640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f26940b4340 legacy=0x7f2694145a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:06.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f26940b4e90 legacy=0x7f269413f5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:06.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.444+0000 7f2699076640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f2694146150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:06.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.645+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f26940b4e90 legacy=0x7f269413f5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:06.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.645+0000 7f2693fff640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f26940b4340 legacy=0x7f2694145a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:06.646 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:06.645+0000 7f2699076640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f2694146150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:07.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:07.046+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f26940b4e90 legacy=0x7f269413f5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:07.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:07.046+0000 7f2693fff640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f26940b4340 legacy=0x7f2694145a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:07.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:07.046+0000 7f2699076640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f2694146150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:07.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:07.847+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f26940b4e90 legacy=0x7f269413f5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:07.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:07.847+0000 7f2693fff640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f26940b4340 legacy=0x7f2694145a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:07.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:07.847+0000 7f2699076640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f2694146150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 10 monclient(hunting): tick 2024-03-25T19:27:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:27:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 1 monclient(hunting): continuing hunt 2024-03-25T19:27:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:27:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f26940b4340 legacy=0x7f2694145a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f2694146150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f26940b4e90 legacy=0x7f269413f5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:09.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-a con 0x7f2678003680 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-b con 0x7f2678007270 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-c con 0x7f269413fd00 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940a96b0 con 0x7f2678003680 2024-03-25T19:27:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940b5270 con 0x7f269413fd00 2024-03-25T19:27:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f2678003680 legacy=0x7f2678005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940ab330 con 0x7f2678007270 2024-03-25T19:27:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f26927fc640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f2693fff640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:09.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.444+0000 7f2699076640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f267800cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:09.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.645+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f2678003680 legacy=0x7f2678005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:09.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.645+0000 7f2693fff640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:09.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:09.645+0000 7f2699076640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f267800cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:10.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:10.046+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f2678003680 legacy=0x7f2678005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:10.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:10.046+0000 7f2693fff640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:10.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:10.046+0000 7f2699076640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f267800cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:10.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:10.847+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f2678003680 legacy=0x7f2678005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:10.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:10.847+0000 7f2693fff640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:10.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:10.847+0000 7f2699076640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f267800cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:12.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 10 monclient(hunting): tick 2024-03-25T19:27:12.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:27:12.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 1 monclient(hunting): continuing hunt 2024-03-25T19:27:12.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:27:12.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780096e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:12.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f267800cf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f2678003680 legacy=0x7f2678005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-a con 0x7f2678003680 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-c con 0x7f269413fd00 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-b con 0x7f2678007270 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940a9d80 con 0x7f2678003680 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940a96b0 con 0x7f269413fd00 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940b5270 con 0x7f2678007270 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f26927fc640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f2678003680 legacy=0x7f2678006200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f2699076640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:12.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.444+0000 7f2693fff640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f2678005a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:12.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.645+0000 7f2699076640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:12.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.645+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f2678003680 legacy=0x7f2678006200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:12.645 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:12.645+0000 7f2693fff640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f2678005a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:13.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:13.046+0000 7f2699076640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:13.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:13.046+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f2678003680 legacy=0x7f2678006200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:13.046 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:13.046+0000 7f2693fff640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f2678005a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:13.300 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:00:06.842 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:13.607 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:00:07.148 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:13.775 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:00:07.374 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:13.777 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:00:07.376 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:13.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:13.847+0000 7f2699076640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780096e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6790/0 2024-03-25T19:27:13.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:13.847+0000 7f2698875640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f2678003680 legacy=0x7f2678006200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.52:6789/0 2024-03-25T19:27:13.847 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:13.847+0000 7f2693fff640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f2678005a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.179:6789/0 2024-03-25T19:27:14.103 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:00:07.701 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:14.103 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:00:07.702 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:15.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 10 monclient(hunting): tick 2024-03-25T19:27:15.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:27:15.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 1 monclient(hunting): continuing hunt 2024-03-25T19:27:15.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:27:15.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 1 -- >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780096e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:15.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 1 -- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f2678005a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:15.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 1 -- >> v1:172.21.15.52:6789/0 conn(0x7f2678003680 legacy=0x7f2678006200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:27:15.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:15.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-a con 0x7f267800cf60 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:15.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-b con 0x7f2678007270 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:15.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-c con 0x7f269413fd00 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:15.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940ab330 con 0x7f267800cf60 2024-03-25T19:27:15.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940a9d80 con 0x7f269413fd00 2024-03-25T19:27:15.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940a96b0 con 0x7f2678007270 2024-03-25T19:27:15.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.444+0000 7f26927fc640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:15.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.501+0000 7f2699076640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 0x7f267800d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:58408/0 (socket says 172.21.15.52:58408) 2024-03-25T19:27:15.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.501+0000 7f2699076640 1 -- 172.21.15.52:0/2400337742 learned_addr learned my addr 172.21.15.52:0/2400337742 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:15.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.782+0000 7f2693fff640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780069b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T19:27:15.782 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.782+0000 7f2693fff640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780069b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:27:15.783 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.782+0000 7f2693fff640 1 --1- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 0x7f26780069b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:27:15.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.815+0000 7f2698875640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6789/0 conn(0x7f267800cf60 legacy=0x7f2678006200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T19:27:15.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.816+0000 7f2698875640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6789/0 conn(0x7f267800cf60 legacy=0x7f2678006200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:27:15.816 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:15.816+0000 7f2698875640 1 --1- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6789/0 conn(0x7f267800cf60 0x7f2678006200 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:27:18.445 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.444+0000 7f26927fc640 10 monclient(hunting): tick 2024-03-25T19:27:18.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.444+0000 7f26927fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:27:18.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.444+0000 7f26927fc640 1 monclient(hunting): continuing hunt 2024-03-25T19:27:18.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.444+0000 7f26927fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:27:18.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.444+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6790/0 conn(0x7f2678007270 legacy=0x7f26780069b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:18.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.444+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.179:6789/0 conn(0x7f269413fd00 legacy=0x7f267800d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:18.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.444+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6789/0 conn(0x7f267800cf60 legacy=0x7f2678006200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:18.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.445+0000 7f26927fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:27:18.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.445+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-b con 0x7f26780096f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:18.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.445+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-c con 0x7f2678005a80 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:18.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.445+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-a con 0x7f267800cf60 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:18.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.445+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940b5270 con 0x7f267800cf60 2024-03-25T19:27:18.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.445+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2678005fd0 con 0x7f2678005a80 2024-03-25T19:27:18.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.445+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2678006a50 con 0x7f26780096f0 2024-03-25T19:27:18.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:18.445+0000 7f26927fc640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:21.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 10 monclient(hunting): tick 2024-03-25T19:27:21.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:27:21.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 1 monclient(hunting): continuing hunt 2024-03-25T19:27:21.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:27:21.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6790/0 conn(0x7f26780096f0 legacy=0x7f267800ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:21.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.179:6789/0 conn(0x7f2678005a80 legacy=0x7f2678009b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:21.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6789/0 conn(0x7f267800cf60 legacy=0x7f2678006200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:21.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:21.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-a con 0x7f267800d380 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:21.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-b con 0x7f267800a950 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:21.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-c con 0x7f2678005a80 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:21.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2678006c30 con 0x7f267800d380 2024-03-25T19:27:21.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2678006200 con 0x7f2678005a80 2024-03-25T19:27:21.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26780063e0 con 0x7f267800a950 2024-03-25T19:27:21.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:21.446+0000 7f26927fc640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:24.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 10 monclient(hunting): tick 2024-03-25T19:27:24.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:27:24.446 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 1 monclient(hunting): continuing hunt 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6790/0 conn(0x7f267800a950 legacy=0x7f2678010760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.179:6789/0 conn(0x7f2678005a80 legacy=0x7f2678009b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6789/0 conn(0x7f267800d380 legacy=0x7f267800a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-b con 0x7f267800a950 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-c con 0x7f267800ded0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 10 monclient(hunting): picked mon.noname-a con 0x7f267800d380 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2678006610 con 0x7f267800d380 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267800a240 con 0x7f267800ded0 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 1 -- 172.21.15.52:0/2400337742 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267800a3d0 con 0x7f267800a950 2024-03-25T19:27:24.447 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.445+0000 7f26927fc640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:24.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.837+0000 7f26937fe640 1 -- 172.21.15.52:0/2400337742 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266412852 0 0) 0x7f2678006610 con 0x7f267800d380 2024-03-25T19:27:24.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.837+0000 7f26937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:27:24.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.837+0000 7f26937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:24.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.837+0000 7f26937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:24.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.837+0000 7f26937fe640 10 monclient(hunting): my global_id is 4104 2024-03-25T19:27:24.838 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.837+0000 7f26937fe640 1 -- 172.21.15.52:0/2400337742 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2684003520 con 0x7f267800d380 2024-03-25T19:27:24.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.883+0000 7f26937fe640 1 -- 172.21.15.52:0/2400337742 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3000639583 0 0) 0x7f2684003520 con 0x7f267800d380 2024-03-25T19:27:24.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.883+0000 7f26937fe640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6790/0 conn(0x7f267800a950 legacy=0x7f267800d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:24.884 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.883+0000 7f26937fe640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.179:6789/0 conn(0x7f267800ded0 legacy=0x7f2678013880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.883+0000 7f26937fe640 10 monclient: _finish_hunting 0 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.883+0000 7f26937fe640 1 monclient: found mon.noname-a 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.883+0000 7f26937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.883+0000 7f26937fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.884+0000 7f26937fe640 1 -- 172.21.15.52:0/2400337742 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267800a560 con 0x7f267800d380 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.884+0000 7f26937fe640 10 monclient: _finish_auth 0 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.884+0000 7f26937fe640 10 monclient: _check_auth_tickets 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.884+0000 7f26937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.884+0000 7f26937fe640 1 -- 172.21.15.52:0/2400337742 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f268c004450 con 0x7f267800d380 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.884+0000 7f26937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.884+0000 7f26937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.884+0000 7f26937fe640 10 monclient: dump: 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-25T19:27:24.885 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:24.886 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:24.886 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:24.886 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:24.886 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:24.886 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:24.886 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:24.886 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:24.886 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:24.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.884+0000 7f269aad9640 5 monclient: authenticate success, global_id 4104 2024-03-25T19:27:24.886 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.884+0000 7f269aad9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.924+0000 7f26937fe640 1 -- 172.21.15.52:0/2400337742 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f268c0045b0 con 0x7f267800d380 2024-03-25T19:27:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.924+0000 7f26937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.924+0000 7f26937fe640 1 -- 172.21.15.52:0/2400337742 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f268c0049c0 con 0x7f267800d380 2024-03-25T19:27:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.924+0000 7f26937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.924+0000 7f26937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.924+0000 7f26937fe640 10 monclient: dump: 2024-03-25T19:27:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-25T19:27:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:24.925 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:24.926 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:24.926 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:24.926 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:24.926 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:24.926 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:24.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.924+0000 7f269aad9640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:24.926 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.924+0000 7f269aad9640 4 set_mon_vals no callback set 2024-03-25T19:27:24.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.925+0000 7f269aad9640 10 monclient: shutdown 2024-03-25T19:27:24.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.925+0000 7f269aad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:24.928 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.925+0000 7f269aad9640 1 -- 172.21.15.52:0/2400337742 >> v1:172.21.15.52:6789/0 conn(0x7f267800d380 legacy=0x7f2678009b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.925+0000 7f269aad9640 1 -- 172.21.15.52:0/2400337742 shutdown_connections 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.925+0000 7f269aad9640 1 -- 172.21.15.52:0/2400337742 >> 172.21.15.52:0/2400337742 conn(0x7f2694066ac0 msgr2=0x7f2694066ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.925+0000 7f269aad9640 1 -- 172.21.15.52:0/2400337742 shutdown_connections 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.925+0000 7f269aad9640 1 -- 172.21.15.52:0/2400337742 wait complete. 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.925+0000 7f269aad9640 10 monclient: build_initial_monmap 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.926+0000 7f269aad9640 10 monclient: monmap: 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:24.929 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.926+0000 7f269aad9640 1 librados: starting msgr at 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.926+0000 7f269aad9640 1 librados: starting objecter 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.926+0000 7f269aad9640 1 Processor -- start 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.926+0000 7f269aad9640 1 -- start start 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.926+0000 7f269aad9640 1 librados: setting wanted keys 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.926+0000 7f269aad9640 1 librados: calling monclient init 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.926+0000 7f269aad9640 10 monclient: init 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f269aad9640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f269aad9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f269aad9640 10 monclient(hunting): picked mon.noname-a con 0x7f267800a950 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f269aad9640 10 monclient(hunting): picked mon.noname-c con 0x7f267800d380 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f269aad9640 10 monclient(hunting): picked mon.noname-b con 0x7f267800ded0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f269aad9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940b77d0 con 0x7f267800a950 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f269aad9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2694138970 con 0x7f267800ded0 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f269aad9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2694141230 con 0x7f267800d380 2024-03-25T19:27:24.930 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f269aad9640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:24.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f269aad9640 10 monclient(hunting): authenticate will time out at 5893.108142s 2024-03-25T19:27:24.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f2693fff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f267800d380 0x7f2694137b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:48032/0 (socket says 172.21.15.52:48032) 2024-03-25T19:27:24.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.927+0000 7f2693fff640 1 -- 172.21.15.52:0/3407473962 learned_addr learned my addr 172.21.15.52:0/3407473962 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:24.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.928+0000 7f26917fa640 10 client.?.objecter ms_handle_connect 0x7f267800d380 2024-03-25T19:27:24.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.928+0000 7f26917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:24.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.928+0000 7f26917fa640 10 client.?.objecter ms_handle_connect 0x7f267800ded0 2024-03-25T19:27:24.931 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.928+0000 7f26917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:24.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.979+0000 7f26917fa640 10 client.?.objecter ms_handle_connect 0x7f267800a950 2024-03-25T19:27:24.979 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.979+0000 7f26917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:24.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.985+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253993167 0 0) 0x7f26940b77d0 con 0x7f267800a950 2024-03-25T19:27:24.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.985+0000 7f26917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:27:24.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.985+0000 7f26917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:24.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.985+0000 7f26917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:24.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.985+0000 7f26917fa640 10 monclient(hunting): my global_id is 4107 2024-03-25T19:27:24.986 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:24.985+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2678006610 con 0x7f267800a950 2024-03-25T19:27:25.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.000+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312152175 0 0) 0x7f2694138970 con 0x7f267800ded0 2024-03-25T19:27:25.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.000+0000 7f26917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:27:25.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.000+0000 7f26917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:25.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.000+0000 7f26917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:25.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.000+0000 7f26917fa640 10 monclient(hunting): my global_id is 4108 2024-03-25T19:27:25.001 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.000+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26940b77d0 con 0x7f267800ded0 2024-03-25T19:27:25.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.003+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532595900 0 0) 0x7f2694141230 con 0x7f267800d380 2024-03-25T19:27:25.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.003+0000 7f26917fa640 10 monclient(hunting): _negotiate 2024-03-25T19:27:25.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.003+0000 7f26917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:25.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.003+0000 7f26917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:25.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.003+0000 7f26917fa640 10 monclient(hunting): my global_id is 4103 2024-03-25T19:27:25.003 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.003+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2694138970 con 0x7f267800d380 2024-03-25T19:27:25.010 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.010+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 902588392 0 0) 0x7f2678006610 con 0x7f267800a950 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.010+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2694141230 con 0x7f267800a950 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.010+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2688004220 con 0x7f267800a950 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.010+0000 7f26917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.010+0000 7f26917fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.010+0000 7f26917fa640 10 monclient(hunting): dump: 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:25.011 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.010+0000 7f26917fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:25.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1147560459 0 0) 0x7f2694141230 con 0x7f267800a950 2024-03-25T19:27:25.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 >> v1:172.21.15.52:6790/0 conn(0x7f267800d380 legacy=0x7f2694137b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:25.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 >> v1:172.21.15.179:6789/0 conn(0x7f267800ded0 legacy=0x7f2694138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:25.038 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f26917fa640 10 monclient: _finish_hunting 0 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f26917fa640 1 monclient: found mon.a 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f26917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f26917fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2694141510 con 0x7f267800a950 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f26917fa640 10 monclient: _finish_auth 0 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f26917fa640 10 monclient: _check_auth_tickets 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f26917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f269aad9640 5 monclient: authenticate success, global_id 4107 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f269aad9640 10 monclient: _renew_subs 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f269aad9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f269aad9640 1 -- 172.21.15.52:0/3407473962 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2694140240 con 0x7f267800a950 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f269aad9640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f269aad9640 10 monclient: _renew_subs 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f269aad9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:25.039 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f269aad9640 1 -- 172.21.15.52:0/3407473962 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26940b7490 con 0x7f267800a950 2024-03-25T19:27:25.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.037+0000 7f269aad9640 1 librados: init done 2024-03-25T19:27:25.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.038+0000 7f269aad9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:25.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.038+0000 7f269aad9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:25.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.038+0000 7f269aad9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:25.040 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.038+0000 7f269aad9640 1 -- 172.21.15.52:0/3407473962 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269404f730 con 0x7f267800a950 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2688004630 con 0x7f267800a950 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2688004a40 con 0x7f267800a950 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 10 monclient: dump: 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:25.259 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2688004ff0 con 0x7f267800a950 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f2692ffd640 4 set_mon_vals no callback set 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 238515652 0 0) 0x7f2688006460 con 0x7f267800a950 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 10 client.4107.objecter ms_dispatch 0x7f2694000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2688006880 con 0x7f267800a950 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:25.260 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.258+0000 7f26917fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:25.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.330+0000 7f269aad9640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:27:25.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.330+0000 7f269aad9640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:27:25.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.330+0000 7f269aad9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:25.331 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.330+0000 7f269aad9640 1 -- 172.21.15.52:0/3407473962 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f269404f730 con 0x7f267800a950 2024-03-25T19:27:25.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.500+0000 7f26917fa640 1 -- 172.21.15.52:0/3407473962 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f2688005380 con 0x7f267800a950 2024-03-25T19:27:25.500 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.500+0000 7f26917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:27:25.501 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.500+0000 7f26917fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:27:25.501 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2024-03-25T19:27:25.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 10 librados: watch_flush enter 2024-03-25T19:27:25.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 10 librados: watch_flush exit 2024-03-25T19:27:25.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T19:27:25.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 10 client.4107.objecter successfully canceled tick 2024-03-25T19:27:25.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 10 monclient: shutdown 2024-03-25T19:27:25.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:25.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 1 -- 172.21.15.52:0/3407473962 >> v1:172.21.15.52:6789/0 conn(0x7f267800a950 legacy=0x7f26941d7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:25.502 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 1 -- 172.21.15.52:0/3407473962 shutdown_connections 2024-03-25T19:27:25.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 1 -- 172.21.15.52:0/3407473962 >> 172.21.15.52:0/3407473962 conn(0x7f2694066ac0 msgr2=0x7f26940b7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:25.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 1 -- 172.21.15.52:0/3407473962 shutdown_connections 2024-03-25T19:27:25.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 1 -- 172.21.15.52:0/3407473962 wait complete. 2024-03-25T19:27:25.503 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:25.501+0000 7f269aad9640 1 librados: shutdown 2024-03-25T19:27:25.516 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:27:25.516 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:27:25.516 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:27:25.516 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:27:25.544 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:27:25.544 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:27:25.565 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:25.564+0000 7f7f97048040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:27:25.591 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:27:25.591 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:27:25.644 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:25.644+0000 7f7f97048040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:27:25.659 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-03-25T19:27:25.659 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:27:25.726 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:27:25.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:27:25.754 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:27:25.754 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:27:25.820 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:25.820+0000 7f217b1db040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:27:25.821 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:27:25.822 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:27:25.876 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:25.876+0000 7f217b1db040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:27:25.888 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-25T19:27:25.888 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:27:25.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 037ba8f0-68fe-4829-a15d-6cfeb9255161 0 2024-03-25T19:27:26.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.096+0000 7fc8f48ba640 10 monclient: get_monmap_and_config 2024-03-25T19:27:26.095 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.096+0000 7fc8f48ba640 10 monclient: build_initial_monmap 2024-03-25T19:27:26.096 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.096+0000 7fc8f48ba640 10 monclient: monmap: 2024-03-25T19:27:26.096 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:26.096 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.096 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:27:26.097040+0000 2024-03-25T19:27:26.096 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:27:26.097040+0000 2024-03-25T19:27:26.096 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:26.096 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.096 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:26.096 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:26.096 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:26.096 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.096+0000 7fc8f48ba640 1 Processor -- start 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.096+0000 7fc8f48ba640 1 -- start start 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.096+0000 7fc8f48ba640 10 monclient: init 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.097+0000 7fc8f48ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.097+0000 7fc8f48ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.097+0000 7fc8f48ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc8f00b4700 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.097+0000 7fc8f48ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc8f0142a20 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.097+0000 7fc8f48ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc8f00a96b0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.097+0000 7fc8f48ba640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f00a9d60 con 0x7fc8f00b4700 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.097+0000 7fc8f48ba640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f013ff10 con 0x7fc8f0142a20 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.097+0000 7fc8f48ba640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f00aa1f0 con 0x7fc8f00a96b0 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.097+0000 7fc8f48ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.097+0000 7fc8f48ba640 10 monclient(hunting): authenticate will time out at 5897.432105s 2024-03-25T19:27:26.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.098+0000 7fc8ee59c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fc8f00b4700 0x7fc8f0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:54974/0 (socket says 172.21.15.179:54974) 2024-03-25T19:27:26.099 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.098+0000 7fc8ee59c640 1 -- 172.21.15.179:0/713472570 learned_addr learned my addr 172.21.15.179:0/713472570 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:26.100 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.100+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818994072 0 0) 0x7fc8f013ff10 con 0x7fc8f0142a20 2024-03-25T19:27:26.100 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.100+0000 7fc8ed59a640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.100 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.100+0000 7fc8ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.100 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.100+0000 7fc8ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.100 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.100+0000 7fc8ed59a640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:27:26.100 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.100+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d8003520 con 0x7fc8f0142a20 2024-03-25T19:27:26.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672007154 0 0) 0x7fc8f00a9d60 con 0x7fc8f00b4700 2024-03-25T19:27:26.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:27:26.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f013ff10 con 0x7fc8f00b4700 2024-03-25T19:27:26.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050513248 0 0) 0x7fc8f00aa1f0 con 0x7fc8f00a96b0 2024-03-25T19:27:26.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 10 monclient(hunting): my global_id is 4109 2024-03-25T19:27:26.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.101+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f00a9d60 con 0x7fc8f00a96b0 2024-03-25T19:27:26.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 332260323 0 0) 0x7fc8f013ff10 con 0x7fc8f00b4700 2024-03-25T19:27:26.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 >> v1:172.21.15.52:6790/0 conn(0x7fc8f00a96b0 legacy=0x7fc8f013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 >> v1:172.21.15.179:6789/0 conn(0x7fc8f0142a20 legacy=0x7fc8f013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 10 monclient: _finish_hunting 0 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 1 monclient: found mon.noname-a 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f00b4ae0 con 0x7fc8f00b4700 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 10 monclient: _finish_auth 0 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 10 monclient: _check_auth_tickets 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc8e4001670 con 0x7fc8f00b4700 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8ed59a640 10 monclient: dump: 2024-03-25T19:27:26.105 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.106 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.106 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.106 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.106 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.106 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.106 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.106 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.106 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.106 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.106 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8f48ba640 5 monclient: authenticate success, global_id 4116 2024-03-25T19:27:26.106 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.104+0000 7fc8f48ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:26.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.106+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8e40017d0 con 0x7fc8f00b4700 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.106+0000 7fc8ed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.106+0000 7fc8ed59a640 1 -- 172.21.15.179:0/713472570 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc8e4001be0 con 0x7fc8f00b4700 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.106+0000 7fc8f48ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.106+0000 7fc8f48ba640 4 set_mon_vals no callback set 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.106+0000 7fc8ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.106+0000 7fc8ed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.106+0000 7fc8ed59a640 10 monclient: dump: 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.108 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.107+0000 7fc8f48ba640 10 monclient: shutdown 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.107+0000 7fc8f48ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.107+0000 7fc8f48ba640 1 -- 172.21.15.179:0/713472570 >> v1:172.21.15.52:6789/0 conn(0x7fc8f00b4700 legacy=0x7fc8f0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.107+0000 7fc8f48ba640 1 -- 172.21.15.179:0/713472570 shutdown_connections 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.107+0000 7fc8f48ba640 1 -- 172.21.15.179:0/713472570 >> 172.21.15.179:0/713472570 conn(0x7fc8f00668e0 msgr2=0x7fc8f0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.107+0000 7fc8f48ba640 1 -- 172.21.15.179:0/713472570 shutdown_connections 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.107+0000 7fc8f48ba640 1 -- 172.21.15.179:0/713472570 wait complete. 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.108+0000 7fc8f48ba640 10 monclient: build_initial_monmap 2024-03-25T19:27:26.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.108+0000 7fc8f48ba640 10 monclient: monmap: 2024-03-25T19:27:26.110 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:26.110 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:26.110 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-25T19:27:26.110 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-25T19:27:26.110 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:26.110 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-25T19:27:26.110 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:26.110 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:26.110 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:26.110 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.108+0000 7fc8f48ba640 1 librados: starting msgr at 2024-03-25T19:27:26.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.108+0000 7fc8f48ba640 1 librados: starting objecter 2024-03-25T19:27:26.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.108+0000 7fc8f48ba640 1 Processor -- start 2024-03-25T19:27:26.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.108+0000 7fc8f48ba640 1 -- start start 2024-03-25T19:27:26.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.108+0000 7fc8f48ba640 1 librados: setting wanted keys 2024-03-25T19:27:26.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.108+0000 7fc8f48ba640 1 librados: calling monclient init 2024-03-25T19:27:26.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.108+0000 7fc8f48ba640 10 monclient: init 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8f48ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8f48ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8f48ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc8f00a96b0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8f48ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc8f00b4700 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8f48ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc8f0142a20 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8f48ba640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f0134c80 con 0x7fc8f00b4700 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8f48ba640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f0136f60 con 0x7fc8f0142a20 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8f48ba640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f0137240 con 0x7fc8f00a96b0 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8f48ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8f48ba640 10 monclient(hunting): authenticate will time out at 5897.443998s 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8edd9b640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fc8f00b4700 0x7fc8f01df060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:54976/0 (socket says 172.21.15.179:54976) 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.109+0000 7fc8edd9b640 1 -- 172.21.15.179:0/911444330 learned_addr learned my addr 172.21.15.179:0/911444330 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.110+0000 7fc8deffd640 10 client.?.objecter ms_handle_connect 0x7fc8f00b4700 2024-03-25T19:27:26.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.110+0000 7fc8deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:26.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.112+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319859260 0 0) 0x7fc8f0134c80 con 0x7fc8f00b4700 2024-03-25T19:27:26.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.112+0000 7fc8deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.112+0000 7fc8deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.112+0000 7fc8deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.112+0000 7fc8deffd640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:27:26.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.112+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d0004eb0 con 0x7fc8f00b4700 2024-03-25T19:27:26.117 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.117+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1153416595 0 0) 0x7fc8d0004eb0 con 0x7fc8f00b4700 2024-03-25T19:27:26.117 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.117+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8f0134c80 con 0x7fc8f00b4700 2024-03-25T19:27:26.117 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.117+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc8e00018f0 con 0x7fc8f00b4700 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.117+0000 7fc8deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.117+0000 7fc8deffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.117+0000 7fc8deffd640 10 monclient(hunting): dump: 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.118 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.117+0000 7fc8deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:26.122 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2068326592 0 0) 0x7fc8f0134c80 con 0x7fc8f00b4700 2024-03-25T19:27:26.122 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 >> v1:172.21.15.52:6790/0 conn(0x7fc8f00a96b0 legacy=0x7fc8f0135940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.122 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 >> v1:172.21.15.179:6789/0 conn(0x7fc8f0142a20 legacy=0x7fc8f01e2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.122 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8deffd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:26.122 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8deffd640 1 monclient: found mon.a 2024-03-25T19:27:26.122 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8deffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f0137520 con 0x7fc8f00b4700 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8deffd640 10 monclient: _finish_auth 0 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8deffd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8f48ba640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8f48ba640 10 monclient: _renew_subs 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8f48ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8f48ba640 1 -- 172.21.15.179:0/911444330 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f0136580 con 0x7fc8f00b4700 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8f48ba640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8f48ba640 10 monclient: _renew_subs 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8f48ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8f48ba640 1 -- 172.21.15.179:0/911444330 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f0136230 con 0x7fc8f00b4700 2024-03-25T19:27:26.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.120+0000 7fc8f48ba640 1 librados: init done 2024-03-25T19:27:26.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.121+0000 7fc8f48ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:26.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.121+0000 7fc8f48ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:26.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.121+0000 7fc8f48ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.121+0000 7fc8f48ba640 1 -- 172.21.15.179:0/911444330 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f004f730 con 0x7fc8f00b4700 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.226+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8e0003610 con 0x7fc8f00b4700 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.226+0000 7fc8deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.226+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc8e00039e0 con 0x7fc8f00b4700 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.226+0000 7fc8deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.226+0000 7fc8deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.226+0000 7fc8deffd640 10 monclient: dump: 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.227 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.226+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc8e0003f90 con 0x7fc8f00b4700 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.226+0000 7fc8ecd99640 4 set_mon_vals no callback set 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.227+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1871917650 0 0) 0x7fc8e0006520 con 0x7fc8f00b4700 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.227+0000 7fc8deffd640 10 client.4119.objecter ms_dispatch 0x7fc8f0000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.227+0000 7fc8deffd640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.227+0000 7fc8deffd640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.227+0000 7fc8deffd640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.227+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc8e0004420 con 0x7fc8f00b4700 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.227+0000 7fc8deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:26.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.227+0000 7fc8deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:26.249 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:26.249+0000 7f7f97048040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:27:26.299 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.299+0000 7fc8f48ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "037ba8f0-68fe-4829-a15d-6cfeb9255161", "id": 0}] 2024-03-25T19:27:26.299 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.299+0000 7fc8f48ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "037ba8f0-68fe-4829-a15d-6cfeb9255161", "id": 0}] 2024-03-25T19:27:26.299 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.299+0000 7fc8f48ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.299 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.299+0000 7fc8f48ba640 1 -- 172.21.15.179:0/911444330 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd new", "uuid": "037ba8f0-68fe-4829-a15d-6cfeb9255161", "id": 0} v 0) v1 -- 0x7fc8f00b30f0 con 0x7fc8f00b4700 2024-03-25T19:27:26.305 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:26.305+0000 7f7f97048040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:27:26.352 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:26.351+0000 7f7f97048040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:27:26.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.407+0000 7fc8deffd640 1 -- 172.21.15.179:0/911444330 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "037ba8f0-68fe-4829-a15d-6cfeb9255161", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1045176435 0 732707515) 0x7fc8e00057c0 con 0x7fc8f00b4700 2024-03-25T19:27:26.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.407+0000 7fc8deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "037ba8f0-68fe-4829-a15d-6cfeb9255161", "id": 0}] 2024-03-25T19:27:26.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.407+0000 7fc8deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:26.407 INFO:teuthology.orchestra.run.smithi179.stdout:0 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 10 librados: watch_flush enter 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 10 librados: watch_flush exit 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 10 monclient: shutdown 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 1 -- 172.21.15.179:0/911444330 >> v1:172.21.15.52:6789/0 conn(0x7fc8f00b4700 legacy=0x7fc8f01df060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 1 -- 172.21.15.179:0/911444330 shutdown_connections 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 1 -- 172.21.15.179:0/911444330 >> 172.21.15.179:0/911444330 conn(0x7fc8f00668e0 msgr2=0x7fc8f00b6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 1 -- 172.21.15.179:0/911444330 shutdown_connections 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 1 -- 172.21.15.179:0/911444330 wait complete. 2024-03-25T19:27:26.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.408+0000 7fc8f48ba640 1 librados: shutdown 2024-03-25T19:27:26.410 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:26.408+0000 7f7f97048040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:27:26.419 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 1c191de0-2061-4c78-97e5-c60bc8ba9c84 1 2024-03-25T19:27:26.429 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:26.429+0000 7f217b1db040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:27:26.455 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:26.454+0000 7f7f97048040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:27:26.486 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:26.486+0000 7f217b1db040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:27:26.520 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.520+0000 7f1ced996640 10 monclient: get_monmap_and_config 2024-03-25T19:27:26.520 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.520+0000 7f1ced996640 10 monclient: build_initial_monmap 2024-03-25T19:27:26.520 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.520+0000 7f1ced996640 10 monclient: monmap: 2024-03-25T19:27:26.520 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:26.520 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.520 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:27:26.521481+0000 2024-03-25T19:27:26.520 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:27:26.521481+0000 2024-03-25T19:27:26.521 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:26.521 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.521 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:26.521 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:26.521 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:26.521 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.522 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 1 Processor -- start 2024-03-25T19:27:26.522 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 1 -- start start 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 10 monclient: init 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce80a9920 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce80b45a0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce8146450 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce806b100 con 0x7f1ce8146450 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce80b2940 con 0x7f1ce80b45a0 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce813fec0 con 0x7f1ce80a9920 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.521+0000 7f1ced996640 10 monclient(hunting): authenticate will time out at 5897.856393s 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.522+0000 7f1ce77fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f1ce8146450 0x7f1ce814dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:54978/0 (socket says 172.21.15.179:54978) 2024-03-25T19:27:26.523 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.522+0000 7f1ce77fe640 1 -- 172.21.15.179:0/1597268904 learned_addr learned my addr 172.21.15.179:0/1597268904 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:26.524 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859970724 0 0) 0x7f1ce80b2940 con 0x7f1ce80b45a0 2024-03-25T19:27:26.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:27:26.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd4003520 con 0x7f1ce80b45a0 2024-03-25T19:27:26.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378184697 0 0) 0x7f1ce806b100 con 0x7f1ce8146450 2024-03-25T19:27:26.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:27:26.525 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.524+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce80b2940 con 0x7f1ce8146450 2024-03-25T19:27:26.526 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.525+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430840684 0 0) 0x7f1ce813fec0 con 0x7f1ce80a9920 2024-03-25T19:27:26.526 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.525+0000 7f1ce5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.526 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.525+0000 7f1ce5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.526 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.525+0000 7f1ce5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.526 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.525+0000 7f1ce5ffb640 10 monclient(hunting): my global_id is 4112 2024-03-25T19:27:26.526 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.525+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce806b100 con 0x7f1ce80a9920 2024-03-25T19:27:26.528 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2841943334 0 0) 0x7f1cd4003520 con 0x7f1ce80b45a0 2024-03-25T19:27:26.528 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 >> v1:172.21.15.52:6790/0 conn(0x7f1ce80a9920 legacy=0x7f1ce813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.528 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 >> v1:172.21.15.52:6789/0 conn(0x7f1ce8146450 legacy=0x7f1ce814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.528 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce8140050 con 0x7f1ce80b45a0 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1cd8001690 con 0x7f1ce80b45a0 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ce5ffb640 10 monclient: dump: 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.529 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.530 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.530 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.530 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.530 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.530 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.530 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.530 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.530 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ced996640 5 monclient: authenticate success, global_id 4117 2024-03-25T19:27:26.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.528+0000 7f1ced996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:26.533 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:26.534+0000 7f217b1db040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:27:26.556 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.556+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cd80017f0 con 0x7f1ce80b45a0 2024-03-25T19:27:26.556 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.556+0000 7f1ce5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:26.556 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.556+0000 7f1ce5ffb640 1 -- 172.21.15.179:0/1597268904 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1cd8001a60 con 0x7f1ce80b45a0 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.556+0000 7f1ced996640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.556+0000 7f1ced996640 4 set_mon_vals no callback set 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.556+0000 7f1ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.556+0000 7f1ce5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.556+0000 7f1ce5ffb640 10 monclient: dump: 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.557 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.557+0000 7f1ced996640 10 monclient: shutdown 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.557+0000 7f1ced996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.557+0000 7f1ced996640 1 -- 172.21.15.179:0/1597268904 >> v1:172.21.15.179:6789/0 conn(0x7f1ce80b45a0 legacy=0x7f1ce8145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.557+0000 7f1ced996640 1 -- 172.21.15.179:0/1597268904 shutdown_connections 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.557+0000 7f1ced996640 1 -- 172.21.15.179:0/1597268904 >> 172.21.15.179:0/1597268904 conn(0x7f1ce80b7f40 msgr2=0x7f1ce80b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.557+0000 7f1ced996640 1 -- 172.21.15.179:0/1597268904 shutdown_connections 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.557+0000 7f1ced996640 1 -- 172.21.15.179:0/1597268904 wait complete. 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.558+0000 7f1ced996640 10 monclient: build_initial_monmap 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.558+0000 7f1ced996640 10 monclient: monmap: 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:26.560 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.558+0000 7f1ced996640 1 librados: starting msgr at 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.558+0000 7f1ced996640 1 librados: starting objecter 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 1 Processor -- start 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 1 -- start start 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 1 librados: setting wanted keys 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 1 librados: calling monclient init 2024-03-25T19:27:26.561 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 10 monclient: init 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce80a9920 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce80b45a0 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce8146450 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce81d8050 con 0x7f1ce80b45a0 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce813cb10 con 0x7f1ce8146450 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce813cca0 con 0x7f1ce80a9920 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.559+0000 7f1ced996640 10 monclient(hunting): authenticate will time out at 5897.894533s 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.560+0000 7f1ce6ffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f1ce80a9920 0x7f1ce81d4770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.179:58052/0 (socket says 172.21.15.179:58052) 2024-03-25T19:27:26.562 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.560+0000 7f1ce6ffd640 1 -- 172.21.15.179:0/3562831429 learned_addr learned my addr 172.21.15.179:0/3562831429 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:26.563 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.561+0000 7f1cdffff640 10 client.?.objecter ms_handle_connect 0x7f1ce80a9920 2024-03-25T19:27:26.563 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.561+0000 7f1cdffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.561+0000 7f1cdffff640 10 client.?.objecter ms_handle_connect 0x7f1ce80b45a0 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.561+0000 7f1cdffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 10 client.?.objecter ms_handle_connect 0x7f1ce8146450 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164652665 0 0) 0x7f1ce81d8050 con 0x7f1ce80b45a0 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc8004eb0 con 0x7f1ce80b45a0 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453189126 0 0) 0x7f1ce813cca0 con 0x7f1ce80a9920 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.564 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.565 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.565 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:27:26.565 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.563+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce81d8050 con 0x7f1ce80a9920 2024-03-25T19:27:26.565 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.565+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196962814 0 0) 0x7f1ce813cb10 con 0x7f1ce8146450 2024-03-25T19:27:26.565 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.565+0000 7f1cdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.565 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.565+0000 7f1cdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.565 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.565+0000 7f1cdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.565 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.565+0000 7f1cdffff640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:27:26.566 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.565+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce813cca0 con 0x7f1ce8146450 2024-03-25T19:27:26.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.568+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 238126440 0 0) 0x7f1cc8004eb0 con 0x7f1ce80b45a0 2024-03-25T19:27:26.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.568+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ce813cb10 con 0x7f1ce80b45a0 2024-03-25T19:27:26.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.568+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1cd8001960 con 0x7f1ce80b45a0 2024-03-25T19:27:26.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.568+0000 7f1cdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.568+0000 7f1cdffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:26.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.568+0000 7f1cdffff640 10 monclient(hunting): dump: 2024-03-25T19:27:26.569 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.569 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.569 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.569 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.569 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.569 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.569 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.569 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.569 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.569 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.569 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.568+0000 7f1cdffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:26.573 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4197273993 0 0) 0x7f1ce813cb10 con 0x7f1ce80b45a0 2024-03-25T19:27:26.573 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 >> v1:172.21.15.52:6790/0 conn(0x7f1ce80a9920 legacy=0x7f1ce81d4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 >> v1:172.21.15.179:6789/0 conn(0x7f1ce8146450 legacy=0x7f1ce81401f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1cdffff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:26.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1cdffff640 1 monclient: found mon.a 2024-03-25T19:27:26.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1cdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:26.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1cdffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.574 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce81d4e80 con 0x7f1ce80b45a0 2024-03-25T19:27:26.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1cdffff640 10 monclient: _finish_auth 0 2024-03-25T19:27:26.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1cdffff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:26.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1cdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:26.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1ced996640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:27:26.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1ced996640 10 monclient: _renew_subs 2024-03-25T19:27:26.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1ced996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.575 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1ced996640 1 -- 172.21.15.179:0/3562831429 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce81d5130 con 0x7f1ce80b45a0 2024-03-25T19:27:26.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1ced996640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:26.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1ced996640 10 monclient: _renew_subs 2024-03-25T19:27:26.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1ced996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1ced996640 1 -- 172.21.15.179:0/3562831429 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce81d7d90 con 0x7f1ce80b45a0 2024-03-25T19:27:26.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.572+0000 7f1ced996640 1 librados: init done 2024-03-25T19:27:26.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.573+0000 7f1ced996640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:26.576 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.573+0000 7f1ced996640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:26.577 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.573+0000 7f1ced996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.577 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.573+0000 7f1ced996640 1 -- 172.21.15.179:0/3562831429 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce804f730 con 0x7f1ce80b45a0 2024-03-25T19:27:26.578 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cd80038a0 con 0x7f1ce80b45a0 2024-03-25T19:27:26.578 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:26.579 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1cd8003bb0 con 0x7f1ce80b45a0 2024-03-25T19:27:26.579 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.579 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:26.579 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 10 monclient: dump: 2024-03-25T19:27:26.579 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.579 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.580 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.580 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.580 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.580 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.580 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.580 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.580 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.581 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.581 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1ce57fa640 4 set_mon_vals no callback set 2024-03-25T19:27:26.581 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1cd80040a0 con 0x7f1ce80b45a0 2024-03-25T19:27:26.581 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1736270943 0 0) 0x7f1cd80054f0 con 0x7f1ce80b45a0 2024-03-25T19:27:26.581 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 10 client.4125.objecter ms_dispatch 0x7f1ce8000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:27:26.581 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:27:26.581 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:27:26.582 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.576+0000 7f1cdffff640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:27:26.588 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:26.587+0000 7f7f97048040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:27:26.590 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:26.590+0000 7f217b1db040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:27:26.632 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:26.632+0000 7f217b1db040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:27:26.661 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:26.661+0000 7f7f97048040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:27:26.677 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.677+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1cd80057f0 con 0x7f1ce80b45a0 2024-03-25T19:27:26.677 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.677+0000 7f1cdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:26.677 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.677+0000 7f1cdffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:26.710 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:26.710+0000 7f7f97048040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:27:26.749 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:26.750+0000 7f217b1db040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:27:26.750 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.750+0000 7f1ced996640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1c191de0-2061-4c78-97e5-c60bc8ba9c84", "id": 1}] 2024-03-25T19:27:26.750 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.750+0000 7f1ced996640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1c191de0-2061-4c78-97e5-c60bc8ba9c84", "id": 1}] 2024-03-25T19:27:26.750 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.750+0000 7f1ced996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.750 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.750+0000 7f1ced996640 1 -- 172.21.15.179:0/3562831429 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1c191de0-2061-4c78-97e5-c60bc8ba9c84", "id": 1} v 0) v1 -- 0x7f1ce8144e00 con 0x7f1ce80b45a0 2024-03-25T19:27:26.765 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:26.764+0000 7f7f97048040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:27:26.787 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.787+0000 7f1cdffff640 1 -- 172.21.15.179:0/3562831429 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1c191de0-2061-4c78-97e5-c60bc8ba9c84", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1530590656 0 940483276) 0x7f1cd8005a30 con 0x7f1ce80b45a0 2024-03-25T19:27:26.787 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.787+0000 7f1cdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1c191de0-2061-4c78-97e5-c60bc8ba9c84", "id": 1}] 2024-03-25T19:27:26.787 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.787+0000 7f1cdffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:26.787 INFO:teuthology.orchestra.run.smithi179.stdout:1 2024-03-25T19:27:26.788 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 10 librados: watch_flush enter 2024-03-25T19:27:26.789 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 10 librados: watch_flush exit 2024-03-25T19:27:26.789 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:27:26.789 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:27:26.789 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 10 monclient: shutdown 2024-03-25T19:27:26.789 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:26.789 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 1 -- 172.21.15.179:0/3562831429 >> v1:172.21.15.52:6789/0 conn(0x7f1ce80b45a0 legacy=0x7f1ce8142a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.789 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 1 -- 172.21.15.179:0/3562831429 shutdown_connections 2024-03-25T19:27:26.789 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 1 -- 172.21.15.179:0/3562831429 >> 172.21.15.179:0/3562831429 conn(0x7f1ce80b7f40 msgr2=0x7f1ce8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:26.789 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 1 -- 172.21.15.179:0/3562831429 shutdown_connections 2024-03-25T19:27:26.789 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 1 -- 172.21.15.179:0/3562831429 wait complete. 2024-03-25T19:27:26.789 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.788+0000 7f1ced996640 1 librados: shutdown 2024-03-25T19:27:26.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 8a2511b0-7d13-4706-be6b-6a51ff228663 2 2024-03-25T19:27:26.818 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:26.818+0000 7f7f97048040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:27:26.822 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:26.822+0000 7f217b1db040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:27:26.871 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:26.871+0000 7f217b1db040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.922+0000 7f901c09a640 10 monclient: get_monmap_and_config 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.922+0000 7f901c09a640 10 monclient: build_initial_monmap 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.922+0000 7f901c09a640 10 monclient: monmap: 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:27:26.923138+0000 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:27:26.923138+0000 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:26.922 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.922+0000 7f901c09a640 1 Processor -- start 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 1 -- start start 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 10 monclient: init 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 10 monclient(hunting): picked mon.noname-a con 0x7f90140668e0 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 10 monclient(hunting): picked mon.noname-c con 0x7f9014066cc0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 10 monclient(hunting): picked mon.noname-b con 0x7f9014142eb0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901406f700 con 0x7f90140668e0 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9014142710 con 0x7f9014066cc0 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901406f970 con 0x7f9014142eb0 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:26.924 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.923+0000 7f901c09a640 10 monclient(hunting): authenticate will time out at 5898.258540s 2024-03-25T19:27:26.925 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.924+0000 7f901a637640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f9014142eb0 0x7f901414db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.179:58054/0 (socket says 172.21.15.179:58054) 2024-03-25T19:27:26.925 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.924+0000 7f901a637640 1 -- 172.21.15.179:0/3253872892 learned_addr learned my addr 172.21.15.179:0/3253872892 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:26.926 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:26.927+0000 7f217b1db040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:27:26.950 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.950+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433486943 0 0) 0x7f901406f700 con 0x7f90140668e0 2024-03-25T19:27:26.950 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.950+0000 7f9018e34640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.951 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.950+0000 7f9018e34640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.951 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.950+0000 7f9018e34640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.951 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.950+0000 7f9018e34640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:27:26.951 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.950+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9000003520 con 0x7f90140668e0 2024-03-25T19:27:26.953 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116130891 0 0) 0x7f9014142710 con 0x7f9014066cc0 2024-03-25T19:27:26.953 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.953 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.953 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.953 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:27:26.953 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f901406f700 con 0x7f9014066cc0 2024-03-25T19:27:26.954 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845707438 0 0) 0x7f901406f970 con 0x7f9014142eb0 2024-03-25T19:27:26.954 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.954 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.954 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.954 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:27:26.954 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.953+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9014142710 con 0x7f9014142eb0 2024-03-25T19:27:26.956 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2348537071 0 0) 0x7f9000003520 con 0x7f90140668e0 2024-03-25T19:27:26.956 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 >> v1:172.21.15.52:6790/0 conn(0x7f9014142eb0 legacy=0x7f901414db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 >> v1:172.21.15.179:6789/0 conn(0x7f9014066cc0 legacy=0x7f90141457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 10 monclient: _finish_hunting 0 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 1 monclient: found mon.noname-a 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9014142990 con 0x7f90140668e0 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 10 monclient: _finish_auth 0 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 10 monclient: _check_auth_tickets 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9010001690 con 0x7f90140668e0 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f901c09a640 5 monclient: authenticate success, global_id 4128 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f901c09a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:26.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f9018e34640 10 monclient: dump: 2024-03-25T19:27:26.958 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.958 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.958 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.958 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.958 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.958 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.958 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.958 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.958 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.958 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.958 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.956+0000 7f901c09a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.960+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90100017f0 con 0x7f90140668e0 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.960+0000 7f9018e34640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.960+0000 7f9018e34640 1 -- 172.21.15.179:0/3253872892 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9010001c00 con 0x7f90140668e0 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.960+0000 7f9018e34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.960+0000 7f9018e34640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.960+0000 7f9018e34640 10 monclient: dump: 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.960 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.961 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.961 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.961 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.961 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.960+0000 7f901c09a640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:26.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.960+0000 7f901c09a640 4 set_mon_vals no callback set 2024-03-25T19:27:26.963 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.961+0000 7f901c09a640 10 monclient: shutdown 2024-03-25T19:27:26.963 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.961+0000 7f901c09a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:26.963 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.961+0000 7f901c09a640 1 -- 172.21.15.179:0/3253872892 >> v1:172.21.15.52:6789/0 conn(0x7f90140668e0 legacy=0x7f9014072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.961+0000 7f901c09a640 1 -- 172.21.15.179:0/3253872892 shutdown_connections 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.961+0000 7f901c09a640 1 -- 172.21.15.179:0/3253872892 >> 172.21.15.179:0/3253872892 conn(0x7f90140b4810 msgr2=0x7f90140b4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.961+0000 7f901c09a640 1 -- 172.21.15.179:0/3253872892 shutdown_connections 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.961+0000 7f901c09a640 1 -- 172.21.15.179:0/3253872892 wait complete. 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.961+0000 7f901c09a640 10 monclient: build_initial_monmap 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.961+0000 7f901c09a640 10 monclient: monmap: 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:26.964 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 1 librados: starting msgr at 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 1 librados: starting objecter 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 1 Processor -- start 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 1 -- start start 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 1 librados: setting wanted keys 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 1 librados: calling monclient init 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 10 monclient: init 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 10 monclient(hunting): picked mon.noname-b con 0x7f90140668e0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 10 monclient(hunting): picked mon.noname-a con 0x7f9014142eb0 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 10 monclient(hunting): picked mon.noname-c con 0x7f901413dee0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:26.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901413f000 con 0x7f9014142eb0 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901413e4e0 con 0x7f90140668e0 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901413f270 con 0x7f901413dee0 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.962+0000 7f901c09a640 10 monclient(hunting): authenticate will time out at 5898.297417s 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.963+0000 7f9019635640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f9014142eb0 0x7f90141d7820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:55010/0 (socket says 172.21.15.179:55010) 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.963+0000 7f9019635640 1 -- 172.21.15.179:0/1973372698 learned_addr learned my addr 172.21.15.179:0/1973372698 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.963+0000 7f900a7fc640 10 client.?.objecter ms_handle_connect 0x7f9014142eb0 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.963+0000 7f900a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.963+0000 7f900a7fc640 10 client.?.objecter ms_handle_connect 0x7f90140668e0 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.963+0000 7f900a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.963+0000 7f900a7fc640 10 client.?.objecter ms_handle_connect 0x7f901413dee0 2024-03-25T19:27:26.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.963+0000 7f900a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:26.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539663846 0 0) 0x7f901413e4e0 con 0x7f90140668e0 2024-03-25T19:27:26.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:27:26.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff8004eb0 con 0x7f90140668e0 2024-03-25T19:27:26.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309209566 0 0) 0x7f901413f270 con 0x7f901413dee0 2024-03-25T19:27:26.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:27:26.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f901413e4e0 con 0x7f901413dee0 2024-03-25T19:27:26.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824009134 0 0) 0x7f901413f000 con 0x7f9014142eb0 2024-03-25T19:27:26.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:26.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:26.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:26.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:27:26.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.965+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f901413f270 con 0x7f9014142eb0 2024-03-25T19:27:26.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 948827656 0 0) 0x7f8ff8004eb0 con 0x7f90140668e0 2024-03-25T19:27:26.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f901413f000 con 0x7f90140668e0 2024-03-25T19:27:26.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9010001b00 con 0x7f90140668e0 2024-03-25T19:27:26.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:26.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 10 monclient(hunting): dump: 2024-03-25T19:27:26.970 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.970 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.970 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1019317642 0 0) 0x7f901413e4e0 con 0x7f901413dee0 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ff8004eb0 con 0x7f901413dee0 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f900c001bf0 con 0x7f901413dee0 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.970+0000 7f900a7fc640 10 monclient(hunting): dump: 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.971 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.972 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.972 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.972 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.972 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.972 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.972 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.972 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.972 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.971+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1425754649 0 0) 0x7f901413f270 con 0x7f9014142eb0 2024-03-25T19:27:26.972 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.971+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f901413e4e0 con 0x7f9014142eb0 2024-03-25T19:27:26.972 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.971+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9004001930 con 0x7f9014142eb0 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.971+0000 7f900a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.971+0000 7f900a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.971+0000 7f900a7fc640 10 monclient(hunting): dump: 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.973 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2060148137 0 0) 0x7f901413e4e0 con 0x7f9014142eb0 2024-03-25T19:27:26.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 >> v1:172.21.15.52:6790/0 conn(0x7f901413dee0 legacy=0x7f90141e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 >> v1:172.21.15.179:6789/0 conn(0x7f90140668e0 legacy=0x7f901413d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f900a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f900a7fc640 1 monclient: found mon.a 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f900a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f900a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90141d7f30 con 0x7f9014142eb0 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f900a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f900a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f900a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f901c09a640 5 monclient: authenticate success, global_id 4131 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f901c09a640 10 monclient: _renew_subs 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f901c09a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f901c09a640 1 -- 172.21.15.179:0/1973372698 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90141d81e0 con 0x7f9014142eb0 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f901c09a640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f901c09a640 10 monclient: _renew_subs 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f901c09a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f901c09a640 1 -- 172.21.15.179:0/1973372698 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90141e38d0 con 0x7f9014142eb0 2024-03-25T19:27:26.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.975+0000 7f901c09a640 1 librados: init done 2024-03-25T19:27:26.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.976+0000 7f901c09a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:26.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.976+0000 7f901c09a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:26.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.976+0000 7f901c09a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:26.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.976+0000 7f901c09a640 1 -- 172.21.15.179:0/1973372698 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90141d7540 con 0x7f9014142eb0 2024-03-25T19:27:26.978 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:26.978+0000 7f217b1db040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:27:26.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9004003610 con 0x7f9014142eb0 2024-03-25T19:27:26.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:26.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9004003880 con 0x7f9014142eb0 2024-03-25T19:27:26.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:26.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 10 monclient: dump: 2024-03-25T19:27:26.980 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9004003e30 con 0x7f9014142eb0 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900b7fe640 4 set_mon_vals no callback set 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 828645473 0 0) 0x7f90040064e0 con 0x7f9014142eb0 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 10 client.4131.objecter ms_dispatch 0x7f9014000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:27:26.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:27:26.982 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:26.980+0000 7f900a7fc640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T19:27:27.079 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.080+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f90040061d0 con 0x7f9014142eb0 2024-03-25T19:27:27.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.080+0000 7f900a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:27.080 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.080+0000 7f900a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:27.152 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.152+0000 7f901c09a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8a2511b0-7d13-4706-be6b-6a51ff228663", "id": 2}] 2024-03-25T19:27:27.152 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.152+0000 7f901c09a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8a2511b0-7d13-4706-be6b-6a51ff228663", "id": 2}] 2024-03-25T19:27:27.152 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.152+0000 7f901c09a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:27.152 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.152+0000 7f901c09a640 1 -- 172.21.15.179:0/1973372698 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8a2511b0-7d13-4706-be6b-6a51ff228663", "id": 2} v 0) v1 -- 0x7f901404f730 con 0x7f9014142eb0 2024-03-25T19:27:27.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.181+0000 7f900a7fc640 1 -- 172.21.15.179:0/1973372698 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8a2511b0-7d13-4706-be6b-6a51ff228663", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1638208732 0 216599125) 0x7f90040040e0 con 0x7f9014142eb0 2024-03-25T19:27:27.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.181+0000 7f900a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8a2511b0-7d13-4706-be6b-6a51ff228663", "id": 2}] 2024-03-25T19:27:27.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.181+0000 7f900a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:27.181 INFO:teuthology.orchestra.run.smithi179.stdout:2 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 10 librados: watch_flush enter 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 10 librados: watch_flush exit 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 10 client.4131.objecter successfully canceled tick 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 10 monclient: shutdown 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 1 -- 172.21.15.179:0/1973372698 >> v1:172.21.15.52:6789/0 conn(0x7f9014142eb0 legacy=0x7f90141d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 1 -- 172.21.15.179:0/1973372698 shutdown_connections 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 1 -- 172.21.15.179:0/1973372698 >> 172.21.15.179:0/1973372698 conn(0x7f90140b4810 msgr2=0x7f90140b4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 1 -- 172.21.15.179:0/1973372698 shutdown_connections 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 1 -- 172.21.15.179:0/1973372698 wait complete. 2024-03-25T19:27:27.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.182+0000 7f901c09a640 1 librados: shutdown 2024-03-25T19:27:27.192 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 0ac59362-fcca-46ec-8c8b-13d7e7829481 3 2024-03-25T19:27:27.249 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:27.249+0000 7f7f97048040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.292+0000 7f24af8a8640 10 monclient: get_monmap_and_config 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.292+0000 7f24af8a8640 10 monclient: build_initial_monmap 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.292+0000 7f24af8a8640 10 monclient: monmap: 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:27:27.293940+0000 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:27:27.293940+0000 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.293+0000 7f24af8a8640 1 Processor -- start 2024-03-25T19:27:27.295 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.293+0000 7f24af8a8640 1 -- start start 2024-03-25T19:27:27.295 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.293+0000 7f24af8a8640 10 monclient: init 2024-03-25T19:27:27.295 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24af8a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:27.295 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24af8a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:27:27.295 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24af8a8640 10 monclient(hunting): picked mon.noname-b con 0x7f24a80b4700 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:27.295 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24af8a8640 10 monclient(hunting): picked mon.noname-c con 0x7f24a8142a20 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:27.295 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24af8a8640 10 monclient(hunting): picked mon.noname-a con 0x7f24a80a96b0 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:27.295 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24af8a8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a80a9d60 con 0x7f24a80a96b0 2024-03-25T19:27:27.295 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24af8a8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a813ff10 con 0x7f24a8142a20 2024-03-25T19:27:27.295 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24af8a8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a80aa1f0 con 0x7f24a80b4700 2024-03-25T19:27:27.295 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24af8a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:27.296 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24af8a8640 10 monclient(hunting): authenticate will time out at 5898.628988s 2024-03-25T19:27:27.296 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24ade45640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f24a80a96b0 0x7f24a813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:55014/0 (socket says 172.21.15.179:55014) 2024-03-25T19:27:27.296 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.294+0000 7f24ade45640 1 -- 172.21.15.179:0/962253156 learned_addr learned my addr 172.21.15.179:0/962253156 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:27.297 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875823469 0 0) 0x7f24a813ff10 con 0x7f24a8142a20 2024-03-25T19:27:27.297 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.297 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.297 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.297 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:27:27.297 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2494003520 con 0x7f24a8142a20 2024-03-25T19:27:27.297 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851952423 0 0) 0x7f24a80a9d60 con 0x7f24a80a96b0 2024-03-25T19:27:27.297 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.297 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.297 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.297 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:27:27.298 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.297+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a813ff10 con 0x7f24a80a96b0 2024-03-25T19:27:27.298 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.298+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180487991 0 0) 0x7f24a80aa1f0 con 0x7f24a80b4700 2024-03-25T19:27:27.298 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.298+0000 7f249ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.298 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.298+0000 7f249ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.298 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.298+0000 7f249ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.298 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.298+0000 7f249ffff640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:27:27.298 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.298+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a80a9d60 con 0x7f24a80b4700 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3568742144 0 0) 0x7f2494003520 con 0x7f24a8142a20 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 >> v1:172.21.15.52:6790/0 conn(0x7f24a80b4700 legacy=0x7f24a8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 >> v1:172.21.15.52:6789/0 conn(0x7f24a80a96b0 legacy=0x7f24a813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 1 monclient: found mon.noname-c 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a80b4ae0 con 0x7f24a8142a20 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 10 monclient: _finish_auth 0 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:27.301 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f24af8a8640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f24a4001690 con 0x7f24a8142a20 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f24af8a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f249ffff640 10 monclient: dump: 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.302 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.303 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.303 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.300+0000 7f24af8a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:27.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.302+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24a40017f0 con 0x7f24a8142a20 2024-03-25T19:27:27.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.302+0000 7f249ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:27.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.302+0000 7f249ffff640 1 -- 172.21.15.179:0/962253156 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f24a4001c40 con 0x7f24a8142a20 2024-03-25T19:27:27.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.302+0000 7f249ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.302+0000 7f249ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:27.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.302+0000 7f249ffff640 10 monclient: dump: 2024-03-25T19:27:27.304 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.304 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.304 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.302+0000 7f24af8a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.302+0000 7f24af8a8640 4 set_mon_vals no callback set 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.303+0000 7f24af8a8640 10 monclient: shutdown 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.303+0000 7f24af8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.303+0000 7f24af8a8640 1 -- 172.21.15.179:0/962253156 >> v1:172.21.15.179:6789/0 conn(0x7f24a8142a20 legacy=0x7f24a813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.303+0000 7f24af8a8640 1 -- 172.21.15.179:0/962253156 shutdown_connections 2024-03-25T19:27:27.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.303+0000 7f24af8a8640 1 -- 172.21.15.179:0/962253156 >> 172.21.15.179:0/962253156 conn(0x7f24a80668e0 msgr2=0x7f24a8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.303+0000 7f24af8a8640 1 -- 172.21.15.179:0/962253156 shutdown_connections 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.303+0000 7f24af8a8640 1 -- 172.21.15.179:0/962253156 wait complete. 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.304+0000 7f24af8a8640 10 monclient: build_initial_monmap 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.304+0000 7f24af8a8640 10 monclient: monmap: 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:27.306 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 1 librados: starting msgr at 2024-03-25T19:27:27.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 1 librados: starting objecter 2024-03-25T19:27:27.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 1 Processor -- start 2024-03-25T19:27:27.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 1 -- start start 2024-03-25T19:27:27.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 1 librados: setting wanted keys 2024-03-25T19:27:27.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 1 librados: calling monclient init 2024-03-25T19:27:27.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 10 monclient: init 2024-03-25T19:27:27.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:27.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:27.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 10 monclient(hunting): picked mon.noname-b con 0x7f24a80a96b0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 10 monclient(hunting): picked mon.noname-c con 0x7f24a80b4700 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 10 monclient(hunting): picked mon.noname-a con 0x7f24a8142a20 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a8139d20 con 0x7f24a8142a20 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a81376c0 con 0x7f24a80a96b0 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a8136050 con 0x7f24a80b4700 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.305+0000 7f24af8a8640 10 monclient(hunting): authenticate will time out at 5898.640292s 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.306+0000 7f24ade45640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f24a8142a20 0x7f24a81e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:55026/0 (socket says 172.21.15.179:55026) 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.306+0000 7f24ade45640 1 -- 172.21.15.179:0/2517107841 learned_addr learned my addr 172.21.15.179:0/2517107841 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.306+0000 7f249dffb640 10 client.?.objecter ms_handle_connect 0x7f24a8142a20 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.306+0000 7f249dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.306+0000 7f249dffb640 10 client.?.objecter ms_handle_connect 0x7f24a80a96b0 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.306+0000 7f249dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.306+0000 7f249dffb640 10 client.?.objecter ms_handle_connect 0x7f24a80b4700 2024-03-25T19:27:27.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.306+0000 7f249dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.307+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682031607 0 0) 0x7f24a8139d20 con 0x7f24a8142a20 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.307+0000 7f249dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.307+0000 7f249dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.307+0000 7f249dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.307+0000 7f249dffb640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.307+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f248c004eb0 con 0x7f24a8142a20 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.308+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017032419 0 0) 0x7f24a81376c0 con 0x7f24a80a96b0 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.308+0000 7f249dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.308+0000 7f249dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.308+0000 7f249dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.308+0000 7f249dffb640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.308+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a8139d20 con 0x7f24a80a96b0 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.309+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395107926 0 0) 0x7f24a8136050 con 0x7f24a80b4700 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.309+0000 7f249dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.309+0000 7f249dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.312 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.309+0000 7f249dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.312 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.309+0000 7f249dffb640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:27:27.312 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.309+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a81376c0 con 0x7f24a80b4700 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.313+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2576392360 0 0) 0x7f24a8139d20 con 0x7f24a80a96b0 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.313+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24a8136050 con 0x7f24a80a96b0 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.313+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2498001870 con 0x7f24a80a96b0 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.313+0000 7f249dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.313+0000 7f249dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.313+0000 7f249dffb640 10 monclient(hunting): dump: 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.313 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.314 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.313+0000 7f249dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:27.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.314+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3270557093 0 0) 0x7f248c004eb0 con 0x7f24a8142a20 2024-03-25T19:27:27.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.314+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24a8139d20 con 0x7f24a8142a20 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.314+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f24a0001bd0 con 0x7f24a8142a20 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.314+0000 7f249dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.314+0000 7f249dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.314+0000 7f249dffb640 10 monclient(hunting): dump: 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.315 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.315+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 940982022 0 0) 0x7f24a81376c0 con 0x7f24a80b4700 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.315+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f248c004eb0 con 0x7f24a80b4700 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.315+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f24a4001b00 con 0x7f24a80b4700 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.315+0000 7f249dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.315+0000 7f249dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.315+0000 7f249dffb640 10 monclient(hunting): dump: 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.317 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4112776574 0 0) 0x7f24a8136050 con 0x7f24a80a96b0 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 >> v1:172.21.15.52:6790/0 conn(0x7f24a80b4700 legacy=0x7f24a8138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 >> v1:172.21.15.52:6789/0 conn(0x7f24a8142a20 legacy=0x7f24a81e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f249dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f249dffb640 1 monclient: found mon.b 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f249dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f249dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a813a850 con 0x7f24a80a96b0 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f249dffb640 10 monclient: _finish_auth 0 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f249dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f249dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f24af8a8640 5 monclient: authenticate success, global_id 4132 2024-03-25T19:27:27.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f24af8a8640 10 monclient: _renew_subs 2024-03-25T19:27:27.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f24af8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:27.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f24af8a8640 1 -- 172.21.15.179:0/2517107841 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a813aaa0 con 0x7f24a80a96b0 2024-03-25T19:27:27.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f24af8a8640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:27.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f24af8a8640 10 monclient: _renew_subs 2024-03-25T19:27:27.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f24af8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:27.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f24af8a8640 1 -- 172.21.15.179:0/2517107841 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a8138a70 con 0x7f24a80a96b0 2024-03-25T19:27:27.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.316+0000 7f24af8a8640 1 librados: init done 2024-03-25T19:27:27.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.317+0000 7f24af8a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:27.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.317+0000 7f24af8a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:27.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.317+0000 7f24af8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:27.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.317+0000 7f24af8a8640 1 -- 172.21.15.179:0/2517107841 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a8135ea0 con 0x7f24a80a96b0 2024-03-25T19:27:27.371 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:27.372+0000 7f217b1db040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:27:27.396 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:27.396+0000 7f7f97048040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:27:27.507 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f217b1db040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2498003610 con 0x7f24a80a96b0 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2498001d80 con 0x7f24a80a96b0 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 10 monclient: dump: 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.508 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249f7fe640 4 set_mon_vals no callback set 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2498003d30 con 0x7f24a80a96b0 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1845264198 0 0) 0x7f2498005380 con 0x7f24a80a96b0 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 10 client.4132.objecter ms_dispatch 0x7f24a8000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2498005780 con 0x7f24a80a96b0 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:27.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.507+0000 7f249dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:27.510 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:27.507+0000 7f7f97048040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:27:27.556 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:27.556+0000 7f7f97048040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:27:27.583 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.583+0000 7f24af8a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0ac59362-fcca-46ec-8c8b-13d7e7829481", "id": 3}] 2024-03-25T19:27:27.583 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.583+0000 7f24af8a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0ac59362-fcca-46ec-8c8b-13d7e7829481", "id": 3}] 2024-03-25T19:27:27.583 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.583+0000 7f24af8a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:27.583 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.583+0000 7f24af8a8640 1 -- 172.21.15.179:0/2517107841 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0ac59362-fcca-46ec-8c8b-13d7e7829481", "id": 3} v 0) v1 -- 0x7f24a8138d20 con 0x7f24a80a96b0 2024-03-25T19:27:27.633 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:27.633+0000 7f217b1db040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:27:27.678 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:27.678+0000 7f7f97048040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:27:27.683 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:27.683+0000 7f217b1db040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:27:27.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.697+0000 7f249dffb640 1 -- 172.21.15.179:0/2517107841 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0ac59362-fcca-46ec-8c8b-13d7e7829481", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2416549009 0 525046306) 0x7f24a8138d20 con 0x7f24a80a96b0 2024-03-25T19:27:27.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.698+0000 7f249dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0ac59362-fcca-46ec-8c8b-13d7e7829481", "id": 3}] 2024-03-25T19:27:27.697 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.698+0000 7f249dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:27.698 INFO:teuthology.orchestra.run.smithi179.stdout:3 2024-03-25T19:27:27.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.698+0000 7f24af8a8640 10 librados: watch_flush enter 2024-03-25T19:27:27.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.699+0000 7f24af8a8640 10 librados: watch_flush exit 2024-03-25T19:27:27.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.699+0000 7f24af8a8640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T19:27:27.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.699+0000 7f24af8a8640 10 client.4132.objecter successfully canceled tick 2024-03-25T19:27:27.699 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.699+0000 7f24af8a8640 10 monclient: shutdown 2024-03-25T19:27:27.700 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.699+0000 7f24af8a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:27.700 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.699+0000 7f24af8a8640 1 -- 172.21.15.179:0/2517107841 >> v1:172.21.15.179:6789/0 conn(0x7f24a80a96b0 legacy=0x7f24a813a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.700 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.699+0000 7f24af8a8640 1 -- 172.21.15.179:0/2517107841 shutdown_connections 2024-03-25T19:27:27.700 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.699+0000 7f24af8a8640 1 -- 172.21.15.179:0/2517107841 >> 172.21.15.179:0/2517107841 conn(0x7f24a80668e0 msgr2=0x7f24a80b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:27.700 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.699+0000 7f24af8a8640 1 -- 172.21.15.179:0/2517107841 shutdown_connections 2024-03-25T19:27:27.700 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.699+0000 7f24af8a8640 1 -- 172.21.15.179:0/2517107841 wait complete. 2024-03-25T19:27:27.700 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.699+0000 7f24af8a8640 1 librados: shutdown 2024-03-25T19:27:27.709 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 2fa32263-00b3-4fe0-85db-58725da804fe 4 2024-03-25T19:27:27.792 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:27.792+0000 7f7f97048040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:27:27.803 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:27.803+0000 7f217b1db040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:27:27.808 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.809+0000 7fb0de4c8640 10 monclient: get_monmap_and_config 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.809+0000 7fb0de4c8640 10 monclient: build_initial_monmap 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.809+0000 7fb0de4c8640 10 monclient: monmap: 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:27:27.810079+0000 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:27:27.810079+0000 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:27.809 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.809+0000 7fb0de4c8640 1 Processor -- start 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.809+0000 7fb0de4c8640 1 -- start start 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.809+0000 7fb0de4c8640 10 monclient: init 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0de4c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0de4c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0de4c8640 10 monclient(hunting): picked mon.noname-a con 0x7fb0d80b4700 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0de4c8640 10 monclient(hunting): picked mon.noname-b con 0x7fb0d8142a20 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0de4c8640 10 monclient(hunting): picked mon.noname-c con 0x7fb0d80a96b0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0de4c8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d80a9d60 con 0x7fb0d80b4700 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0de4c8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d813ff10 con 0x7fb0d80a96b0 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0de4c8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d80aa1f0 con 0x7fb0d8142a20 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0de4c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0de4c8640 10 monclient(hunting): authenticate will time out at 5899.145034s 2024-03-25T19:27:27.811 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0d7fff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fb0d80b4700 0x7fb0d8142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:55042/0 (socket says 172.21.15.179:55042) 2024-03-25T19:27:27.812 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.810+0000 7fb0d7fff640 1 -- 172.21.15.179:0/3100567622 learned_addr learned my addr 172.21.15.179:0/3100567622 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:27.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927772466 0 0) 0x7fb0d813ff10 con 0x7fb0d80a96b0 2024-03-25T19:27:27.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:27:27.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c0003520 con 0x7fb0d80a96b0 2024-03-25T19:27:27.813 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924582518 0 0) 0x7fb0d80a9d60 con 0x7fb0d80b4700 2024-03-25T19:27:27.814 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.814 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.814 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.814 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:27:27.814 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.813+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d813ff10 con 0x7fb0d80b4700 2024-03-25T19:27:27.814 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.814+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941825120 0 0) 0x7fb0d80aa1f0 con 0x7fb0d8142a20 2024-03-25T19:27:27.814 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.814+0000 7fb0d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.814 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.814+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.814+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.814+0000 7fb0d6ffd640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:27:27.815 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.814+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d80a9d60 con 0x7fb0d8142a20 2024-03-25T19:27:27.816 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3142902950 0 0) 0x7fb0d813ff10 con 0x7fb0d80b4700 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 >> v1:172.21.15.52:6790/0 conn(0x7fb0d8142a20 legacy=0x7fb0d813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 >> v1:172.21.15.179:6789/0 conn(0x7fb0d80a96b0 legacy=0x7fb0d813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d80b4ae0 con 0x7fb0d80b4700 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb0c8001670 con 0x7fb0d80b4700 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:27.817 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0d6ffd640 10 monclient: dump: 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0de4c8640 5 monclient: authenticate success, global_id 4140 2024-03-25T19:27:27.818 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.816+0000 7fb0de4c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:27.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.818+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0c80017d0 con 0x7fb0d80b4700 2024-03-25T19:27:27.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.818+0000 7fb0d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:27.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.818+0000 7fb0de4c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:27.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.818+0000 7fb0de4c8640 4 set_mon_vals no callback set 2024-03-25T19:27:27.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.818+0000 7fb0d6ffd640 1 -- 172.21.15.179:0/3100567622 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb0c8001be0 con 0x7fb0d80b4700 2024-03-25T19:27:27.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.818+0000 7fb0d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.818+0000 7fb0d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:27.820 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.818+0000 7fb0d6ffd640 10 monclient: dump: 2024-03-25T19:27:27.820 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.820 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.819+0000 7fb0de4c8640 10 monclient: shutdown 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.819+0000 7fb0de4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.819+0000 7fb0de4c8640 1 -- 172.21.15.179:0/3100567622 >> v1:172.21.15.52:6789/0 conn(0x7fb0d80b4700 legacy=0x7fb0d8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.819+0000 7fb0de4c8640 1 -- 172.21.15.179:0/3100567622 shutdown_connections 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.819+0000 7fb0de4c8640 1 -- 172.21.15.179:0/3100567622 >> 172.21.15.179:0/3100567622 conn(0x7fb0d80668e0 msgr2=0x7fb0d8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:27.821 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.819+0000 7fb0de4c8640 1 -- 172.21.15.179:0/3100567622 shutdown_connections 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.819+0000 7fb0de4c8640 1 -- 172.21.15.179:0/3100567622 wait complete. 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.820+0000 7fb0de4c8640 10 monclient: build_initial_monmap 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.820+0000 7fb0de4c8640 10 monclient: monmap: 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.822 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.820+0000 7fb0de4c8640 1 librados: starting msgr at 2024-03-25T19:27:27.823 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.820+0000 7fb0de4c8640 1 librados: starting objecter 2024-03-25T19:27:27.823 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.820+0000 7fb0de4c8640 1 Processor -- start 2024-03-25T19:27:27.825 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 1 -- start start 2024-03-25T19:27:27.825 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 1 librados: setting wanted keys 2024-03-25T19:27:27.825 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 1 librados: calling monclient init 2024-03-25T19:27:27.825 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 10 monclient: init 2024-03-25T19:27:27.825 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:27.825 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:27.825 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 10 monclient(hunting): picked mon.noname-c con 0x7fb0d80a96b0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:27.825 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 10 monclient(hunting): picked mon.noname-a con 0x7fb0d80b4700 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 10 monclient(hunting): picked mon.noname-b con 0x7fb0d8142a20 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d81368a0 con 0x7fb0d80b4700 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d813ee60 con 0x7fb0d8142a20 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d8133740 con 0x7fb0d80a96b0 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.821+0000 7fb0de4c8640 10 monclient(hunting): authenticate will time out at 5899.156591s 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.822+0000 7fb0d77fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fb0d80b4700 0x7fb0d81df060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:55050/0 (socket says 172.21.15.179:55050) 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.822+0000 7fb0d77fe640 1 -- 172.21.15.179:0/117868042 learned_addr learned my addr 172.21.15.179:0/117868042 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.822+0000 7fb0d4ff9640 10 client.?.objecter ms_handle_connect 0x7fb0d80b4700 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.822+0000 7fb0d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.822+0000 7fb0d4ff9640 10 client.?.objecter ms_handle_connect 0x7fb0d80a96b0 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.822+0000 7fb0d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.824+0000 7fb0d4ff9640 10 client.?.objecter ms_handle_connect 0x7fb0d8142a20 2024-03-25T19:27:27.826 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.824+0000 7fb0d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.824+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776501983 0 0) 0x7fb0d81368a0 con 0x7fb0d80b4700 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.824+0000 7fb0d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.824+0000 7fb0d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.824+0000 7fb0d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.824+0000 7fb0d4ff9640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.824+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c4004eb0 con 0x7fb0d80b4700 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.825+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244056206 0 0) 0x7fb0d8133740 con 0x7fb0d80a96b0 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.825+0000 7fb0d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.825+0000 7fb0d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.825+0000 7fb0d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.825+0000 7fb0d4ff9640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:27:27.827 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.825+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d81368a0 con 0x7fb0d80a96b0 2024-03-25T19:27:27.828 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.826+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606634284 0 0) 0x7fb0d813ee60 con 0x7fb0d8142a20 2024-03-25T19:27:27.828 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.826+0000 7fb0d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:27.828 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.826+0000 7fb0d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:27.828 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.826+0000 7fb0d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:27.828 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.826+0000 7fb0d4ff9640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:27:27.828 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.826+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d8133740 con 0x7fb0d8142a20 2024-03-25T19:27:27.829 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.829+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 824626863 0 0) 0x7fb0c4004eb0 con 0x7fb0d80b4700 2024-03-25T19:27:27.829 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.829+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0d813ee60 con 0x7fb0d80b4700 2024-03-25T19:27:27.829 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.829+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb0cc001a30 con 0x7fb0d80b4700 2024-03-25T19:27:27.829 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.829+0000 7fb0d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.829 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.829+0000 7fb0d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:27.829 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.829+0000 7fb0d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:27.829 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.830 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.830 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.830 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.830 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.830 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.830 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.830 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.830 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.830 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.830 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.829+0000 7fb0d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:27.831 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.831+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2279232626 0 0) 0x7fb0d8133740 con 0x7fb0d8142a20 2024-03-25T19:27:27.831 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.831+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0c4004eb0 con 0x7fb0d8142a20 2024-03-25T19:27:27.831 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.831+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb0d0001cd0 con 0x7fb0d8142a20 2024-03-25T19:27:27.831 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.831+0000 7fb0d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.831 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.831+0000 7fb0d4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:27.831 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.831+0000 7fb0d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:27.831 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.831 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.831 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.831 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.831 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.832 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.832 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.832 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.832 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.832 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.833 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2987717575 0 0) 0x7fb0d81368a0 con 0x7fb0d80a96b0 2024-03-25T19:27:27.833 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0d8133740 con 0x7fb0d80a96b0 2024-03-25T19:27:27.833 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb0c8001ae0 con 0x7fb0d80a96b0 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.834 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3569965750 0 0) 0x7fb0d813ee60 con 0x7fb0d80b4700 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 >> v1:172.21.15.52:6790/0 conn(0x7fb0d80a96b0 legacy=0x7fb0d8133e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 >> v1:172.21.15.179:6789/0 conn(0x7fb0d8142a20 legacy=0x7fb0d81e2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 1 monclient: found mon.a 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d8133a20 con 0x7fb0d80b4700 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0de4c8640 5 monclient: authenticate success, global_id 4143 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0de4c8640 10 monclient: _renew_subs 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0de4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0de4c8640 1 -- 172.21.15.179:0/117868042 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d81327a0 con 0x7fb0d80b4700 2024-03-25T19:27:27.835 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0de4c8640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:27.836 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0de4c8640 10 monclient: _renew_subs 2024-03-25T19:27:27.836 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0de4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:27.836 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0de4c8640 1 -- 172.21.15.179:0/117868042 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d812c580 con 0x7fb0d80b4700 2024-03-25T19:27:27.836 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.832+0000 7fb0de4c8640 1 librados: init done 2024-03-25T19:27:27.836 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.833+0000 7fb0de4c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:27.836 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.833+0000 7fb0de4c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:27.836 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.833+0000 7fb0de4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:27.836 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.833+0000 7fb0de4c8640 1 -- 172.21.15.179:0/117868042 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d81335a0 con 0x7fb0d80b4700 2024-03-25T19:27:27.837 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0cc001b90 con 0x7fb0d80b4700 2024-03-25T19:27:27.837 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:27.837 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb0cc0038c0 con 0x7fb0d80b4700 2024-03-25T19:27:27.837 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:27.837 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:27.837 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 10 monclient: dump: 2024-03-25T19:27:27.837 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:27.837 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:27.837 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb0cc003e70 con 0x7fb0d80b4700 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2595892889 0 0) 0x7fb0cc006550 con 0x7fb0d80b4700 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 10 client.4143.objecter ms_dispatch 0x7fb0d8000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d67fc640 4 set_mon_vals no callback set 2024-03-25T19:27:27.838 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.836+0000 7fb0d4ff9640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T19:27:27.900 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:27.900+0000 7f217b1db040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:27:27.941 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.941+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb0cc006150 con 0x7fb0d80b4700 2024-03-25T19:27:27.941 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.941+0000 7fb0d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:27.941 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:27.941+0000 7fb0d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:27.969 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:27.968+0000 7f7f97048040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:27:28.014 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.014+0000 7fb0de4c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2fa32263-00b3-4fe0-85db-58725da804fe", "id": 4}] 2024-03-25T19:27:28.014 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.014+0000 7fb0de4c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2fa32263-00b3-4fe0-85db-58725da804fe", "id": 4}] 2024-03-25T19:27:28.014 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.014+0000 7fb0de4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:28.014 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.014+0000 7fb0de4c8640 1 -- 172.21.15.179:0/117868042 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2fa32263-00b3-4fe0-85db-58725da804fe", "id": 4} v 0) v1 -- 0x7fb0d8132990 con 0x7fb0d80b4700 2024-03-25T19:27:28.021 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:28.021+0000 7f7f97048040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:27:28.044 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.044+0000 7fb0d4ff9640 1 -- 172.21.15.179:0/117868042 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2fa32263-00b3-4fe0-85db-58725da804fe", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1885075108 0 1697012583) 0x7fb0cc0062f0 con 0x7fb0d80b4700 2024-03-25T19:27:28.044 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.044+0000 7fb0d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2fa32263-00b3-4fe0-85db-58725da804fe", "id": 4}] 2024-03-25T19:27:28.044 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.044+0000 7fb0d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:28.044 INFO:teuthology.orchestra.run.smithi179.stdout:4 2024-03-25T19:27:28.045 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 10 librados: watch_flush enter 2024-03-25T19:27:28.045 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 10 librados: watch_flush exit 2024-03-25T19:27:28.045 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T19:27:28.045 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 10 client.4143.objecter successfully canceled tick 2024-03-25T19:27:28.046 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 10 monclient: shutdown 2024-03-25T19:27:28.046 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:28.046 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 1 -- 172.21.15.179:0/117868042 >> v1:172.21.15.52:6789/0 conn(0x7fb0d80b4700 legacy=0x7fb0d81df060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.046 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 1 -- 172.21.15.179:0/117868042 shutdown_connections 2024-03-25T19:27:28.046 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 1 -- 172.21.15.179:0/117868042 >> 172.21.15.179:0/117868042 conn(0x7fb0d80668e0 msgr2=0x7fb0d80b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:28.046 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 1 -- 172.21.15.179:0/117868042 shutdown_connections 2024-03-25T19:27:28.046 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 1 -- 172.21.15.179:0/117868042 wait complete. 2024-03-25T19:27:28.046 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.045+0000 7fb0de4c8640 1 librados: shutdown 2024-03-25T19:27:28.055 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 25acb8c3-a73f-4fbb-8b96-840b236c0235 5 2024-03-25T19:27:28.061 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:28.061+0000 7f217b1db040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:27:28.106 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:28.107+0000 7f217b1db040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:27:28.155 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.155+0000 7f7bd28d9640 10 monclient: get_monmap_and_config 2024-03-25T19:27:28.155 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.155+0000 7f7bd28d9640 10 monclient: build_initial_monmap 2024-03-25T19:27:28.155 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.155+0000 7f7bd28d9640 10 monclient: monmap: 2024-03-25T19:27:28.155 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:28.155 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.155 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:27:28.156278+0000 2024-03-25T19:27:28.155 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:27:28.156278+0000 2024-03-25T19:27:28.155 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:28.155 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.155 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:28.155 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:28.156 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:28.156 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 1 Processor -- start 2024-03-25T19:27:28.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 1 -- start start 2024-03-25T19:27:28.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 10 monclient: init 2024-03-25T19:27:28.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:28.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:28.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 10 monclient(hunting): picked mon.noname-b con 0x7f7bcc0b4700 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:28.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 10 monclient(hunting): picked mon.noname-a con 0x7f7bcc142a20 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:28.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 10 monclient(hunting): picked mon.noname-c con 0x7f7bcc0a96b0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:28.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bcc0a9d60 con 0x7f7bcc142a20 2024-03-25T19:27:28.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bcc13ff10 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bcc0aa1f0 con 0x7f7bcc0b4700 2024-03-25T19:27:28.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:28.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.156+0000 7f7bd28d9640 10 monclient(hunting): authenticate will time out at 5899.491348s 2024-03-25T19:27:28.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.157+0000 7f7bcb7fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f7bcc142a20 0x7f7bcc13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:55066/0 (socket says 172.21.15.179:55066) 2024-03-25T19:27:28.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.157+0000 7f7bcb7fe640 1 -- 172.21.15.179:0/1001495257 learned_addr learned my addr 172.21.15.179:0/1001495257 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:28.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051288887 0 0) 0x7f7bcc13ff10 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:27:28.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb4003520 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525841271 0 0) 0x7f7bcc0a9d60 con 0x7f7bcc142a20 2024-03-25T19:27:28.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:27:28.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.159+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bcc13ff10 con 0x7f7bcc142a20 2024-03-25T19:27:28.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.160+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284448514 0 0) 0x7f7bcc0aa1f0 con 0x7f7bcc0b4700 2024-03-25T19:27:28.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.160+0000 7f7bcaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.160+0000 7f7bcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.160+0000 7f7bcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.160+0000 7f7bcaffd640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:27:28.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.160+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bcc0a9d60 con 0x7f7bcc0b4700 2024-03-25T19:27:28.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 169452803 0 0) 0x7f7bb4003520 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 >> v1:172.21.15.52:6790/0 conn(0x7f7bcc0b4700 legacy=0x7f7bcc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 >> v1:172.21.15.52:6789/0 conn(0x7f7bcc142a20 legacy=0x7f7bcc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:28.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 1 monclient: found mon.noname-c 2024-03-25T19:27:28.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:28.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc0b4ae0 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 10 monclient: _finish_auth 0 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7bc4001a10 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bd28d9640 5 monclient: authenticate success, global_id 4141 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bd28d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bcaffd640 10 monclient: dump: 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.164 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:28.165 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.165 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:28.165 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:28.165 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:28.165 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.162+0000 7f7bd28d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:28.168 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.164+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bc4001b70 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.168 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.164+0000 7f7bcaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:28.168 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.164+0000 7f7bd28d9640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:28.168 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.164+0000 7f7bcaffd640 1 -- 172.21.15.179:0/1001495257 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7bc4001f80 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.168 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.164+0000 7f7bcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.164+0000 7f7bd28d9640 4 set_mon_vals no callback set 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.164+0000 7f7bcaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.164+0000 7f7bcaffd640 10 monclient: dump: 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:28.169 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.165+0000 7f7bd28d9640 10 monclient: shutdown 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.165+0000 7f7bd28d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.165+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1001495257 >> v1:172.21.15.179:6789/0 conn(0x7f7bcc0a96b0 legacy=0x7f7bcc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.165+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1001495257 shutdown_connections 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.165+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1001495257 >> 172.21.15.179:0/1001495257 conn(0x7f7bcc0668e0 msgr2=0x7f7bcc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.165+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1001495257 shutdown_connections 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.165+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1001495257 wait complete. 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.165+0000 7f7bd28d9640 10 monclient: build_initial_monmap 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.165+0000 7f7bd28d9640 10 monclient: monmap: 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-25T19:27:28.170 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.166+0000 7f7bd28d9640 1 librados: starting msgr at 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.166+0000 7f7bd28d9640 1 librados: starting objecter 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.166+0000 7f7bd28d9640 1 Processor -- start 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.166+0000 7f7bd28d9640 1 -- start start 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.166+0000 7f7bd28d9640 1 librados: setting wanted keys 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.166+0000 7f7bd28d9640 1 librados: calling monclient init 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.166+0000 7f7bd28d9640 10 monclient: init 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bd28d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bd28d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bd28d9640 10 monclient(hunting): picked mon.noname-a con 0x7f7bcc0a96b0 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bd28d9640 10 monclient(hunting): picked mon.noname-c con 0x7f7bcc0b4700 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:28.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bd28d9640 10 monclient(hunting): picked mon.noname-b con 0x7f7bcc142a20 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bd28d9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bcc0ab970 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bd28d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bcc13c050 con 0x7f7bcc142a20 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bd28d9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bcc1d81c0 con 0x7f7bcc0b4700 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bd28d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bd28d9640 10 monclient(hunting): authenticate will time out at 5899.501829s 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bcbfff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f7bcc0a96b0 0x7f7bcc13b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:55074/0 (socket says 172.21.15.179:55074) 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bcbfff640 1 -- 172.21.15.179:0/1640265400 learned_addr learned my addr 172.21.15.179:0/1640265400 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bc8ff9640 10 client.?.objecter ms_handle_connect 0x7f7bcc0a96b0 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bc8ff9640 10 client.?.objecter ms_handle_connect 0x7f7bcc142a20 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.167+0000 7f7bc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.168+0000 7f7bc8ff9640 10 client.?.objecter ms_handle_connect 0x7f7bcc0b4700 2024-03-25T19:27:28.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.168+0000 7f7bc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:28.173 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760057978 0 0) 0x7f7bcc13c050 con 0x7f7bcc142a20 2024-03-25T19:27:28.173 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.173 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.173 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb8004eb0 con 0x7f7bcc142a20 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239492726 0 0) 0x7f7bcc0ab970 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.169+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bcc13c050 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.171+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212279647 0 0) 0x7f7bcc1d81c0 con 0x7f7bcc0b4700 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.171+0000 7f7bc8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.171+0000 7f7bc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.171+0000 7f7bc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.171+0000 7f7bc8ff9640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:27:28.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.171+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bcc0ab970 con 0x7f7bcc0b4700 2024-03-25T19:27:28.175 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235333849 0 0) 0x7f7bb8004eb0 con 0x7f7bcc142a20 2024-03-25T19:27:28.175 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bcc1d81c0 con 0x7f7bcc142a20 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7bc4001e80 con 0x7f7bcc142a20 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:28.176 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1784155935 0 0) 0x7f7bcc13c050 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bb8004eb0 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7bbc0019f0 con 0x7f7bcc0a96b0 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.174+0000 7f7bc8ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.177 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:28.178 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:28.178 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:28.178 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.179 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2540441558 0 0) 0x7f7bcc1d81c0 con 0x7f7bcc142a20 2024-03-25T19:27:28.179 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 >> v1:172.21.15.52:6790/0 conn(0x7f7bcc0b4700 legacy=0x7f7bcc1d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.179 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 >> v1:172.21.15.52:6789/0 conn(0x7f7bcc0a96b0 legacy=0x7f7bcc13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.179 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bc8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:27:28.179 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bc8ff9640 1 monclient: found mon.b 2024-03-25T19:27:28.179 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bc8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:28.179 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bc8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc1d7590 con 0x7f7bcc142a20 2024-03-25T19:27:28.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bc8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:27:28.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bc8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:27:28.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:28.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bd28d9640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:27:28.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bd28d9640 10 monclient: _renew_subs 2024-03-25T19:27:28.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bd28d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1640265400 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bcc1d4e90 con 0x7f7bcc142a20 2024-03-25T19:27:28.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bd28d9640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:28.182 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bd28d9640 10 monclient: _renew_subs 2024-03-25T19:27:28.182 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bd28d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.182 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1640265400 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bcc13ae10 con 0x7f7bcc142a20 2024-03-25T19:27:28.182 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.177+0000 7f7bd28d9640 1 librados: init done 2024-03-25T19:27:28.182 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.179+0000 7f7bd28d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:28.182 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.179+0000 7f7bd28d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:28.182 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.179+0000 7f7bd28d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.182 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.179+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1640265400 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bcc1d7720 con 0x7f7bcc142a20 2024-03-25T19:27:28.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bc4002020 con 0x7f7bcc142a20 2024-03-25T19:27:28.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:28.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bca7fc640 4 set_mon_vals no callback set 2024-03-25T19:27:28.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7bc4003dc0 con 0x7f7bcc142a20 2024-03-25T19:27:28.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:28.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 10 monclient: dump: 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7bc40042b0 con 0x7f7bcc142a20 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 990980501 0 0) 0x7f7bc4005a60 con 0x7f7bcc142a20 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 10 client.4144.objecter ms_dispatch 0x7f7bcc000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:27:28.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:27:28.185 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.181+0000 7f7bc8ff9640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T19:27:28.204 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:28.204+0000 7f7f97048040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:27:28.269 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:28.269+0000 7f217b1db040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:27:28.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.278+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7bc4005590 con 0x7f7bcc142a20 2024-03-25T19:27:28.277 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.278+0000 7f7bc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:28.278 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.278+0000 7f7bc8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:28.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.367+0000 7f7bd28d9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "25acb8c3-a73f-4fbb-8b96-840b236c0235", "id": 5}] 2024-03-25T19:27:28.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.367+0000 7f7bd28d9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "25acb8c3-a73f-4fbb-8b96-840b236c0235", "id": 5}] 2024-03-25T19:27:28.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.367+0000 7f7bd28d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.367+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1640265400 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "25acb8c3-a73f-4fbb-8b96-840b236c0235", "id": 5} v 0) v1 -- 0x7f7bcc1d7380 con 0x7f7bcc142a20 2024-03-25T19:27:28.451 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.451+0000 7f7bc8ff9640 1 -- 172.21.15.179:0/1640265400 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "25acb8c3-a73f-4fbb-8b96-840b236c0235", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1753651724 0 1988412176) 0x7f7bc4004560 con 0x7f7bcc142a20 2024-03-25T19:27:28.451 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.451+0000 7f7bc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "25acb8c3-a73f-4fbb-8b96-840b236c0235", "id": 5}] 2024-03-25T19:27:28.451 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.451+0000 7f7bc8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:28.452 INFO:teuthology.orchestra.run.smithi179.stdout:5 2024-03-25T19:27:28.453 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 10 librados: watch_flush enter 2024-03-25T19:27:28.453 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 10 librados: watch_flush exit 2024-03-25T19:27:28.453 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T19:27:28.453 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 10 client.4144.objecter successfully canceled tick 2024-03-25T19:27:28.453 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 10 monclient: shutdown 2024-03-25T19:27:28.453 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:28.453 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1640265400 >> v1:172.21.15.179:6789/0 conn(0x7f7bcc142a20 legacy=0x7f7bcc1d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.454 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1640265400 shutdown_connections 2024-03-25T19:27:28.454 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1640265400 >> 172.21.15.179:0/1640265400 conn(0x7f7bcc0668e0 msgr2=0x7f7bcc13e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:28.454 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1640265400 shutdown_connections 2024-03-25T19:27:28.454 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 1 -- 172.21.15.179:0/1640265400 wait complete. 2024-03-25T19:27:28.454 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.452+0000 7f7bd28d9640 1 librados: shutdown 2024-03-25T19:27:28.463 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new a846d437-978e-4458-a2dd-254e5eb4275c 6 2024-03-25T19:27:28.603 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.603+0000 7f4b38cba640 10 monclient: get_monmap_and_config 2024-03-25T19:27:28.603 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.603+0000 7f4b38cba640 10 monclient: build_initial_monmap 2024-03-25T19:27:28.604 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.603+0000 7f4b38cba640 10 monclient: monmap: 2024-03-25T19:27:28.604 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:28.604 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.604 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:27:28.604899+0000 2024-03-25T19:27:28.604 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:27:28.604899+0000 2024-03-25T19:27:28.604 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:28.604 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.604 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:28.604 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:28.604 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:28.604 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.606 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.604+0000 7f4b38cba640 1 Processor -- start 2024-03-25T19:27:28.606 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.604+0000 7f4b38cba640 1 -- start start 2024-03-25T19:27:28.606 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.604+0000 7f4b38cba640 10 monclient: init 2024-03-25T19:27:28.606 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b38cba640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:28.606 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b38cba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:28.607 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b38cba640 10 monclient(hunting): picked mon.noname-a con 0x7f4b340a9920 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:28.607 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b38cba640 10 monclient(hunting): picked mon.noname-c con 0x7f4b340b45a0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:28.607 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b38cba640 10 monclient(hunting): picked mon.noname-b con 0x7f4b34146450 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:28.607 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b38cba640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3406b100 con 0x7f4b340a9920 2024-03-25T19:27:28.607 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b38cba640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b340b2940 con 0x7f4b340b45a0 2024-03-25T19:27:28.607 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b38cba640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3413fec0 con 0x7f4b34146450 2024-03-25T19:27:28.607 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b38cba640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:28.607 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b38cba640 10 monclient(hunting): authenticate will time out at 5899.939976s 2024-03-25T19:27:28.607 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b3259c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f4b340a9920 0x7f4b3413f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:55082/0 (socket says 172.21.15.179:55082) 2024-03-25T19:27:28.607 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.605+0000 7f4b3259c640 1 -- 172.21.15.179:0/3623282182 learned_addr learned my addr 172.21.15.179:0/3623282182 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:28.608 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708023860 0 0) 0x7f4b340b2940 con 0x7f4b340b45a0 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b20003520 con 0x7f4b340b45a0 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635671078 0 0) 0x7f4b3406b100 con 0x7f4b340a9920 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b340b2940 con 0x7f4b340a9920 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086088120 0 0) 0x7f4b3413fec0 con 0x7f4b34146450 2024-03-25T19:27:28.609 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.610 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.610 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.610 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:27:28.610 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.608+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3406b100 con 0x7f4b34146450 2024-03-25T19:27:28.611 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2870546548 0 0) 0x7f4b20003520 con 0x7f4b340b45a0 2024-03-25T19:27:28.611 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 >> v1:172.21.15.52:6790/0 conn(0x7f4b34146450 legacy=0x7f4b3414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 >> v1:172.21.15.52:6789/0 conn(0x7f4b340a9920 legacy=0x7f4b3413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 10 monclient: _finish_hunting 0 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 1 monclient: found mon.noname-c 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b34140050 con 0x7f4b340b45a0 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 10 monclient: _finish_auth 0 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 10 monclient: _check_auth_tickets 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4b24001690 con 0x7f4b340b45a0 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b38cba640 5 monclient: authenticate success, global_id 4147 2024-03-25T19:27:28.612 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b38cba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b3159a640 10 monclient: dump: 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.613 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.611+0000 7f4b38cba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:28.616 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b240017f0 con 0x7f4b340b45a0 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b3159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b3159a640 1 -- 172.21.15.179:0/3623282182 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4b24001a60 con 0x7f4b340b45a0 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b3159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b3159a640 10 monclient: dump: 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:28.617 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b38cba640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b38cba640 4 set_mon_vals no callback set 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b38cba640 10 monclient: shutdown 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b38cba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b38cba640 1 -- 172.21.15.179:0/3623282182 >> v1:172.21.15.179:6789/0 conn(0x7f4b340b45a0 legacy=0x7f4b34145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b38cba640 1 -- 172.21.15.179:0/3623282182 shutdown_connections 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b38cba640 1 -- 172.21.15.179:0/3623282182 >> 172.21.15.179:0/3623282182 conn(0x7f4b340b7f40 msgr2=0x7f4b340b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b38cba640 1 -- 172.21.15.179:0/3623282182 shutdown_connections 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.613+0000 7f4b38cba640 1 -- 172.21.15.179:0/3623282182 wait complete. 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.614+0000 7f4b38cba640 10 monclient: build_initial_monmap 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.614+0000 7f4b38cba640 10 monclient: monmap: 2024-03-25T19:27:28.618 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.614+0000 7f4b38cba640 1 librados: starting msgr at 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.614+0000 7f4b38cba640 1 librados: starting objecter 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 1 Processor -- start 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 1 -- start start 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 1 librados: setting wanted keys 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 1 librados: calling monclient init 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 10 monclient: init 2024-03-25T19:27:28.619 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 10 monclient(hunting): picked mon.noname-b con 0x7f4b340a9920 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 10 monclient(hunting): picked mon.noname-c con 0x7f4b340b45a0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 10 monclient(hunting): picked mon.noname-a con 0x7f4b34146450 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b341442f0 con 0x7f4b34146450 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b34142bf0 con 0x7f4b340a9920 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b341357d0 con 0x7f4b340b45a0 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.615+0000 7f4b38cba640 10 monclient(hunting): authenticate will time out at 5899.950276s 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.616+0000 7f4b32d9d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f4b34146450 0x7f4b341d70c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:55090/0 (socket says 172.21.15.179:55090) 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.616+0000 7f4b32d9d640 1 -- 172.21.15.179:0/2099336647 learned_addr learned my addr 172.21.15.179:0/2099336647 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.616+0000 7f4b2affd640 10 client.?.objecter ms_handle_connect 0x7f4b340a9920 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.616+0000 7f4b2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.616+0000 7f4b2affd640 10 client.?.objecter ms_handle_connect 0x7f4b34146450 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.616+0000 7f4b2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.616+0000 7f4b2affd640 10 client.?.objecter ms_handle_connect 0x7f4b340b45a0 2024-03-25T19:27:28.620 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.616+0000 7f4b2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:28.621 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949147831 0 0) 0x7f4b34142bf0 con 0x7f4b340a9920 2024-03-25T19:27:28.621 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.621 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b14004eb0 con 0x7f4b340a9920 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183719663 0 0) 0x7f4b341442f0 con 0x7f4b34146450 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.618+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b34142bf0 con 0x7f4b34146450 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.619+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562318998 0 0) 0x7f4b341357d0 con 0x7f4b340b45a0 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.619+0000 7f4b2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.619+0000 7f4b2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:28.622 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.619+0000 7f4b2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:28.623 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.619+0000 7f4b2affd640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:27:28.623 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.619+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b341442f0 con 0x7f4b340b45a0 2024-03-25T19:27:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.622+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3701417668 0 0) 0x7f4b14004eb0 con 0x7f4b340a9920 2024-03-25T19:27:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b341357d0 con 0x7f4b340a9920 2024-03-25T19:27:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4b1c001970 con 0x7f4b340a9920 2024-03-25T19:27:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 10 monclient(hunting): dump: 2024-03-25T19:27:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.624 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1111002905 0 0) 0x7f4b34142bf0 con 0x7f4b34146450 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b14004eb0 con 0x7f4b34146450 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4b2c001c50 con 0x7f4b34146450 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.623+0000 7f4b2affd640 10 monclient(hunting): dump: 2024-03-25T19:27:28.625 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:28.626 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.626 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.626 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.626 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:28.626 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.626 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:28.626 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:28.626 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:28.626 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1337533987 0 0) 0x7f4b341357d0 con 0x7f4b340a9920 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 >> v1:172.21.15.52:6790/0 conn(0x7f4b340b45a0 legacy=0x7f4b341350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 >> v1:172.21.15.52:6789/0 conn(0x7f4b34146450 legacy=0x7f4b341d70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b2affd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b2affd640 1 monclient: found mon.b 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b2affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b34135960 con 0x7f4b340a9920 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b2affd640 10 monclient: _finish_auth 0 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b2affd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b38cba640 5 monclient: authenticate success, global_id 4150 2024-03-25T19:27:28.628 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b38cba640 10 monclient: _renew_subs 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b38cba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b38cba640 1 -- 172.21.15.179:0/2099336647 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3413c1a0 con 0x7f4b340a9920 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b38cba640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b38cba640 10 monclient: _renew_subs 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b38cba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b38cba640 1 -- 172.21.15.179:0/2099336647 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b341d77d0 con 0x7f4b340a9920 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.626+0000 7f4b38cba640 1 librados: init done 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.627+0000 7f4b38cba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.627+0000 7f4b38cba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.627+0000 7f4b38cba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.627+0000 7f4b38cba640 1 -- 172.21.15.179:0/2099336647 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b34142a30 con 0x7f4b340a9920 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.627+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b1c0047e0 con 0x7f4b340a9920 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.628+0000 7f4b2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:28.629 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.628+0000 7f4b30d99640 4 set_mon_vals no callback set 2024-03-25T19:27:28.725 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4b1c004af0 con 0x7f4b340a9920 2024-03-25T19:27:28.725 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:28.725 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:28.725 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 10 monclient: dump: 2024-03-25T19:27:28.725 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:28.725 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.725 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.725 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4b1c005060 con 0x7f4b340a9920 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 545593898 0 0) 0x7f4b1c006930 con 0x7f4b340a9920 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 10 client.4150.objecter ms_dispatch 0x7f4b34000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4b1c018dc0 con 0x7f4b340a9920 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:28.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.724+0000 7f4b2affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:28.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.797+0000 7f4b38cba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a846d437-978e-4458-a2dd-254e5eb4275c", "id": 6}] 2024-03-25T19:27:28.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.797+0000 7f4b38cba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a846d437-978e-4458-a2dd-254e5eb4275c", "id": 6}] 2024-03-25T19:27:28.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.797+0000 7f4b38cba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:28.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.797+0000 7f4b38cba640 1 -- 172.21.15.179:0/2099336647 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a846d437-978e-4458-a2dd-254e5eb4275c", "id": 6} v 0) v1 -- 0x7f4b34142d80 con 0x7f4b340a9920 2024-03-25T19:27:28.865 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.865+0000 7f4b2affd640 1 -- 172.21.15.179:0/2099336647 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a846d437-978e-4458-a2dd-254e5eb4275c", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1351523554 0 1113811849) 0x7f4b1c006410 con 0x7f4b340a9920 2024-03-25T19:27:28.865 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.865+0000 7f4b2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a846d437-978e-4458-a2dd-254e5eb4275c", "id": 6}] 2024-03-25T19:27:28.865 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.865+0000 7f4b2affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:28.866 INFO:teuthology.orchestra.run.smithi179.stdout:6 2024-03-25T19:27:28.867 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 10 librados: watch_flush enter 2024-03-25T19:27:28.867 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 10 librados: watch_flush exit 2024-03-25T19:27:28.867 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T19:27:28.867 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 10 client.4150.objecter successfully canceled tick 2024-03-25T19:27:28.867 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 10 monclient: shutdown 2024-03-25T19:27:28.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:28.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 1 -- 172.21.15.179:0/2099336647 >> v1:172.21.15.179:6789/0 conn(0x7f4b340a9920 legacy=0x7f4b34142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:28.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 1 -- 172.21.15.179:0/2099336647 shutdown_connections 2024-03-25T19:27:28.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 1 -- 172.21.15.179:0/2099336647 >> 172.21.15.179:0/2099336647 conn(0x7f4b340b7f40 msgr2=0x7f4b34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:28.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 1 -- 172.21.15.179:0/2099336647 shutdown_connections 2024-03-25T19:27:28.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 1 -- 172.21.15.179:0/2099336647 wait complete. 2024-03-25T19:27:28.868 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.866+0000 7f4b38cba640 1 librados: shutdown 2024-03-25T19:27:28.877 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 0550ddb7-e774-4908-8042-591d2ea0875f 7 2024-03-25T19:27:28.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.977+0000 7f3338e89640 10 monclient: get_monmap_and_config 2024-03-25T19:27:28.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.977+0000 7f3338e89640 10 monclient: build_initial_monmap 2024-03-25T19:27:28.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.977+0000 7f3338e89640 10 monclient: monmap: 2024-03-25T19:27:28.977 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:28.978 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:28.978 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:27:28.978507+0000 2024-03-25T19:27:28.978 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:27:28.978507+0000 2024-03-25T19:27:28.978 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:28.978 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:28.978 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:28.978 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:28.978 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:28.978 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:28.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 1 Processor -- start 2024-03-25T19:27:28.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 1 -- start start 2024-03-25T19:27:28.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 10 monclient: init 2024-03-25T19:27:28.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:28.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:28.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 10 monclient(hunting): picked mon.noname-a con 0x7f3334070210 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:28.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 10 monclient(hunting): picked mon.noname-b con 0x7f33340b06d0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:28.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 10 monclient(hunting): picked mon.noname-c con 0x7f33340705f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:28.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33340b0b80 con 0x7f3334070210 2024-03-25T19:27:28.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33340b0430 con 0x7f33340705f0 2024-03-25T19:27:28.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3334071670 con 0x7f33340b06d0 2024-03-25T19:27:28.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:28.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.978+0000 7f3338e89640 10 monclient(hunting): authenticate will time out at 5900.313582s 2024-03-25T19:27:28.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.980+0000 7f333259c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f3334070210 0x7f3334140930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.179:55096/0 (socket says 172.21.15.179:55096) 2024-03-25T19:27:28.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:28.980+0000 7f333259c640 1 -- 172.21.15.179:0/4266082317 learned_addr learned my addr 172.21.15.179:0/4266082317 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:29.006 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.007+0000 7f3331d9b640 1 -- 172.21.15.179:0/4266082317 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162241720 0 0) 0x7f332c001490 con 0x7f33340705f0 2024-03-25T19:27:29.007 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.007+0000 7f3331d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:27:29.007 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.007+0000 7f3331d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:29.007 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.007+0000 7f3331d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:29.007 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.007+0000 7f3331d9b640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:27:29.007 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.007+0000 7f3331d9b640 1 -- 172.21.15.179:0/4266082317 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3320003520 con 0x7f33340705f0 2024-03-25T19:27:29.008 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.008+0000 7f3331d9b640 1 -- 172.21.15.179:0/4266082317 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221511190 0 0) 0x7f33340b0b80 con 0x7f3334070210 2024-03-25T19:27:29.008 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.008+0000 7f3331d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:27:29.008 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.008+0000 7f3331d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:29.008 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.008+0000 7f3331d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:29.008 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.008+0000 7f3331d9b640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:27:29.008 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.008+0000 7f3331d9b640 1 -- 172.21.15.179:0/4266082317 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f332c001490 con 0x7f3334070210 2024-03-25T19:27:29.010 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 1 -- 172.21.15.179:0/4266082317 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3581845558 0 0) 0x7f3320003520 con 0x7f33340705f0 2024-03-25T19:27:29.010 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 1 -- 172.21.15.179:0/4266082317 >> v1:172.21.15.52:6790/0 conn(0x7f33340b06d0 legacy=0x7f33340afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.010 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 1 -- 172.21.15.179:0/4266082317 >> v1:172.21.15.52:6789/0 conn(0x7f3334070210 legacy=0x7f3334140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.010 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 1 monclient: found mon.noname-c 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 1 -- 172.21.15.179:0/4266082317 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334070a60 con 0x7f33340705f0 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 10 monclient: _finish_auth 0 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 1 -- 172.21.15.179:0/4266082317 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f332c001a00 con 0x7f33340705f0 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3331d9b640 10 monclient: dump: 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.011 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.012 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.012 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:29.012 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:29.012 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:29.012 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:29.012 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:29.012 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:29.012 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3338e89640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:27:29.012 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.010+0000 7f3338e89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:29.015 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3331d9b640 1 -- 172.21.15.179:0/4266082317 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f332c001b60 con 0x7f33340705f0 2024-03-25T19:27:29.015 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3331d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:29.015 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3331d9b640 1 -- 172.21.15.179:0/4266082317 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f332c001ef0 con 0x7f33340705f0 2024-03-25T19:27:29.015 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3338e89640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3338e89640 4 set_mon_vals no callback set 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3331d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3331d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3331d9b640 10 monclient: dump: 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:29.016 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3338e89640 10 monclient: shutdown 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3338e89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3338e89640 1 -- 172.21.15.179:0/4266082317 >> v1:172.21.15.179:6789/0 conn(0x7f33340705f0 legacy=0x7f333414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3338e89640 1 -- 172.21.15.179:0/4266082317 shutdown_connections 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.012+0000 7f3338e89640 1 -- 172.21.15.179:0/4266082317 >> 172.21.15.179:0/4266082317 conn(0x7f33340b5cd0 msgr2=0x7f33340b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.013+0000 7f3338e89640 1 -- 172.21.15.179:0/4266082317 shutdown_connections 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.013+0000 7f3338e89640 1 -- 172.21.15.179:0/4266082317 wait complete. 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.013+0000 7f3338e89640 10 monclient: build_initial_monmap 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.013+0000 7f3338e89640 10 monclient: monmap: 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:29.017 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 1 librados: starting msgr at 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 1 librados: starting objecter 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 1 Processor -- start 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 1 -- start start 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 1 librados: setting wanted keys 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 1 librados: calling monclient init 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 10 monclient: init 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 10 monclient(hunting): picked mon.noname-b con 0x7f3334070210 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 10 monclient(hunting): picked mon.noname-a con 0x7f33340705f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:29.018 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 10 monclient(hunting): picked mon.noname-c con 0x7f33340b06d0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:29.019 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3334139470 con 0x7f33340705f0 2024-03-25T19:27:29.019 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3334138de0 con 0x7f3334070210 2024-03-25T19:27:29.019 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3334139050 con 0x7f33340b06d0 2024-03-25T19:27:29.019 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:29.019 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.014+0000 7f3338e89640 10 monclient(hunting): authenticate will time out at 5900.349453s 2024-03-25T19:27:29.019 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.015+0000 7f333259c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f3334070210 0x7f333406d740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35978/0 (socket says 172.21.15.179:35978) 2024-03-25T19:27:29.019 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.015+0000 7f333259c640 1 -- 172.21.15.179:0/3344992946 learned_addr learned my addr 172.21.15.179:0/3344992946 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-25T19:27:29.019 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.015+0000 7f332affd640 10 client.?.objecter ms_handle_connect 0x7f3334070210 2024-03-25T19:27:29.019 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.015+0000 7f332affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:29.019 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.015+0000 7f332affd640 10 client.?.objecter ms_handle_connect 0x7f33340705f0 2024-03-25T19:27:29.019 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.015+0000 7f332affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:29.020 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.016+0000 7f332affd640 10 client.?.objecter ms_handle_connect 0x7f33340b06d0 2024-03-25T19:27:29.020 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.016+0000 7f332affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:29.020 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662747071 0 0) 0x7f3334139470 con 0x7f33340705f0 2024-03-25T19:27:29.020 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:29.020 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:29.020 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:29.020 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:27:29.021 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3314004eb0 con 0x7f33340705f0 2024-03-25T19:27:29.021 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029569924 0 0) 0x7f3334138de0 con 0x7f3334070210 2024-03-25T19:27:29.021 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:29.021 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:29.021 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:29.021 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:27:29.021 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.018+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3334139470 con 0x7f3334070210 2024-03-25T19:27:29.022 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2031272303 0 0) 0x7f3314004eb0 con 0x7f33340705f0 2024-03-25T19:27:29.022 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3334138de0 con 0x7f33340705f0 2024-03-25T19:27:29.022 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f331c001820 con 0x7f33340705f0 2024-03-25T19:27:29.022 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:29.022 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:29.022 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 10 monclient(hunting): dump: 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290371785 0 0) 0x7f3334139050 con 0x7f33340b06d0 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:29.023 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:29.024 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:27:29.024 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.022+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3314004eb0 con 0x7f33340b06d0 2024-03-25T19:27:29.024 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.024+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2428533201 0 0) 0x7f3334139470 con 0x7f3334070210 2024-03-25T19:27:29.024 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.024+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3334139050 con 0x7f3334070210 2024-03-25T19:27:29.024 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.025+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f3324001930 con 0x7f3334070210 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.025+0000 7f332affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.025+0000 7f332affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.025+0000 7f332affd640 10 monclient(hunting): dump: 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:29.025 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:29.027 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1811558250 0 0) 0x7f3334138de0 con 0x7f33340705f0 2024-03-25T19:27:29.027 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 >> v1:172.21.15.52:6790/0 conn(0x7f33340b06d0 legacy=0x7f33341d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.027 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 >> v1:172.21.15.179:6789/0 conn(0x7f3334070210 legacy=0x7f333406d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.027 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f332affd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f332affd640 1 monclient: found mon.a 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f332affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f332affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33341396a0 con 0x7f33340705f0 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f332affd640 10 monclient: _finish_auth 0 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f332affd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f332affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f3338e89640 5 monclient: authenticate success, global_id 4161 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f3338e89640 10 monclient: _renew_subs 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f3338e89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f3338e89640 1 -- 172.21.15.179:0/3344992946 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33341d4bc0 con 0x7f33340705f0 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f3338e89640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f3338e89640 10 monclient: _renew_subs 2024-03-25T19:27:29.028 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f3338e89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:29.029 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f3338e89640 1 -- 172.21.15.179:0/3344992946 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f333406cca0 con 0x7f33340705f0 2024-03-25T19:27:29.029 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.027+0000 7f3338e89640 1 librados: init done 2024-03-25T19:27:29.029 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.028+0000 7f3338e89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:29.029 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.028+0000 7f3338e89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:29.029 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.028+0000 7f3338e89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:29.029 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.028+0000 7f3338e89640 1 -- 172.21.15.179:0/3344992946 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f333406cf50 con 0x7f33340705f0 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.031+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f331c001d30 con 0x7f33340705f0 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.031+0000 7f332affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.031+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f331c003790 con 0x7f33340705f0 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.031+0000 7f332affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.031+0000 7f332affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.031+0000 7f332affd640 10 monclient: dump: 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 3 2024-03-25T19:27:29.032 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:29.033 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:29.033 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:29.033 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:27:29.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.031+0000 7f333159a640 4 set_mon_vals no callback set 2024-03-25T19:27:29.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.031+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f331c003d40 con 0x7f33340705f0 2024-03-25T19:27:29.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.031+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 568215923 0 0) 0x7f331c005560 con 0x7f33340705f0 2024-03-25T19:27:29.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.031+0000 7f332affd640 10 client.4161.objecter ms_dispatch 0x7f3334000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:27:29.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.031+0000 7f332affd640 3 client.4161.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:27:29.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.032+0000 7f332affd640 3 client.4161.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:27:29.033 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.032+0000 7f332affd640 20 client.4161.objecter dump_active .. 0 homeless 2024-03-25T19:27:29.091 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:29.091+0000 7f7f97048040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:27:29.130 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.130+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f331c005010 con 0x7f33340705f0 2024-03-25T19:27:29.130 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.130+0000 7f332affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:29.130 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.130+0000 7f332affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:29.145 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:29.146+0000 7f217b1db040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:27:29.202 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.202+0000 7f3338e89640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0550ddb7-e774-4908-8042-591d2ea0875f", "id": 7}] 2024-03-25T19:27:29.202 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.202+0000 7f3338e89640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0550ddb7-e774-4908-8042-591d2ea0875f", "id": 7}] 2024-03-25T19:27:29.203 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.202+0000 7f3338e89640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:29.203 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.202+0000 7f3338e89640 1 -- 172.21.15.179:0/3344992946 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0550ddb7-e774-4908-8042-591d2ea0875f", "id": 7} v 0) v1 -- 0x7f333404f730 con 0x7f33340705f0 2024-03-25T19:27:29.233 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.234+0000 7f332affd640 1 -- 172.21.15.179:0/3344992946 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0550ddb7-e774-4908-8042-591d2ea0875f", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 149059684 0 1371665406) 0x7f331c005390 con 0x7f33340705f0 2024-03-25T19:27:29.234 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.234+0000 7f332affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0550ddb7-e774-4908-8042-591d2ea0875f", "id": 7}] 2024-03-25T19:27:29.234 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.234+0000 7f332affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:29.234 INFO:teuthology.orchestra.run.smithi179.stdout:7 2024-03-25T19:27:29.235 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 10 librados: watch_flush enter 2024-03-25T19:27:29.235 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 10 librados: watch_flush exit 2024-03-25T19:27:29.235 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 20 client.4161.objecter shutdown clearing up homeless session... 2024-03-25T19:27:29.235 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 10 client.4161.objecter successfully canceled tick 2024-03-25T19:27:29.236 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 10 monclient: shutdown 2024-03-25T19:27:29.236 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:29.236 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 1 -- 172.21.15.179:0/3344992946 >> v1:172.21.15.52:6789/0 conn(0x7f33340705f0 legacy=0x7f33341da630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.236 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 1 -- 172.21.15.179:0/3344992946 shutdown_connections 2024-03-25T19:27:29.236 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 1 -- 172.21.15.179:0/3344992946 >> 172.21.15.179:0/3344992946 conn(0x7f33340b5cd0 msgr2=0x7f33340b4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:29.236 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 1 -- 172.21.15.179:0/3344992946 shutdown_connections 2024-03-25T19:27:29.236 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 1 -- 172.21.15.179:0/3344992946 wait complete. 2024-03-25T19:27:29.236 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-25T19:27:29.235+0000 7f3338e89640 1 librados: shutdown 2024-03-25T19:27:29.245 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:27:29.245 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:27:29.246 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:27:29.248 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:27:29.248 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:27:29.248 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:27:29.248 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:27:29.250 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:27:29.251 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:27:29.251 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:27:29.251 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:27:29.253 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:27:29.253 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:27:29.253 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:27:29.253 DEBUG:teuthology.orchestra.run.smithi052:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:27:29.256 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:27:29.256 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:27:29.256 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:27:29.256 DEBUG:teuthology.orchestra.run.smithi179:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:27:29.258 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:27:29.258 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:27:29.259 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:27:29.259 DEBUG:teuthology.orchestra.run.smithi179:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:27:29.261 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:27:29.261 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:27:29.261 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:27:29.261 DEBUG:teuthology.orchestra.run.smithi179:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:27:29.263 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:27:29.263 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:27:29.263 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:27:29.263 DEBUG:teuthology.orchestra.run.smithi179:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:27:29.265 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:27:29.266 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:29.399 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.400+0000 7f2dd5d46640 10 monclient: get_monmap_and_config 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.400+0000 7f2dd5d46640 10 monclient: build_initial_monmap 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.400+0000 7f2dd5d46640 10 monclient: monmap: 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:27:29.400869+0000 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:27:29.400869+0000 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:29.400 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:29.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.401+0000 7f2dd5d46640 1 Processor -- start 2024-03-25T19:27:29.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.401+0000 7f2dd5d46640 1 -- start start 2024-03-25T19:27:29.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.401+0000 7f2dd5d46640 10 monclient: init 2024-03-25T19:27:29.401 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.402+0000 7f2dd5d46640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:29.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.402+0000 7f2dd5d46640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:29.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.402+0000 7f2dd5d46640 10 monclient(hunting): picked mon.noname-a con 0x7f2dd00b4700 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:29.402 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.402+0000 7f2dd5d46640 10 monclient(hunting): picked mon.noname-c con 0x7f2dd0142a20 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:29.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.403+0000 7f2dcf7fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f2dd00b4700 0x7f2dd0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:55826/0 (socket says 172.21.15.52:55826) 2024-03-25T19:27:29.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.403+0000 7f2dcf7fe640 1 -- 172.21.15.52:0/1867743502 learned_addr learned my addr 172.21.15.52:0/1867743502 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:29.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.403+0000 7f2dd5d46640 10 monclient(hunting): picked mon.noname-b con 0x7f2dd00a96b0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:29.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.403+0000 7f2dd5d46640 1 -- 172.21.15.52:0/1867743502 --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd00a9d60 con 0x7f2dd00b4700 2024-03-25T19:27:29.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.403+0000 7f2dd5d46640 1 -- 172.21.15.52:0/1867743502 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd013ff10 con 0x7f2dd0142a20 2024-03-25T19:27:29.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.403+0000 7f2dd5d46640 1 -- 172.21.15.52:0/1867743502 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd00aa1f0 con 0x7f2dd00a96b0 2024-03-25T19:27:29.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.403+0000 7f2dd5d46640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:29.403 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.403+0000 7f2dd5d46640 10 monclient(hunting): authenticate will time out at 5897.583553s 2024-03-25T19:27:29.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.406+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795832234 0 0) 0x7f2dd013ff10 con 0x7f2dd0142a20 2024-03-25T19:27:29.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.406+0000 7f2dce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:29.406 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.406+0000 7f2dce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:29.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.406+0000 7f2dce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:29.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.406+0000 7f2dce7fc640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:27:29.407 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.406+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db8003520 con 0x7f2dd0142a20 2024-03-25T19:27:29.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948807226 0 0) 0x7f2dd00aa1f0 con 0x7f2dd00a96b0 2024-03-25T19:27:29.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:29.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:29.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:29.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:27:29.408 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd013ff10 con 0x7f2dd00a96b0 2024-03-25T19:27:29.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619744885 0 0) 0x7f2dd00a9d60 con 0x7f2dd00b4700 2024-03-25T19:27:29.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:29.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:29.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:29.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 10 monclient(hunting): my global_id is 4164 2024-03-25T19:27:29.409 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.408+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd00aa1f0 con 0x7f2dd00b4700 2024-03-25T19:27:29.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 77016705 0 0) 0x7f2db8003520 con 0x7f2dd0142a20 2024-03-25T19:27:29.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 >> v1:172.21.15.52:6790/0 conn(0x7f2dd00a96b0 legacy=0x7f2dd013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.414 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 >> v1:172.21.15.52:6789/0 conn(0x7f2dd00b4700 legacy=0x7f2dd0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 1 monclient: found mon.noname-c 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd00b4ae0 con 0x7f2dd0142a20 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2dc4001690 con 0x7f2dd0142a20 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.413+0000 7f2dce7fc640 10 monclient: dump: 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.415 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.416 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.416 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:29.416 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:29.416 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:29.416 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:29.416 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:29.416 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:29.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.414+0000 7f2dd5d46640 5 monclient: authenticate success, global_id 4159 2024-03-25T19:27:29.416 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.414+0000 7f2dd5d46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.415+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc40017f0 con 0x7f2dd0142a20 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.415+0000 7f2dce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.415+0000 7f2dd5d46640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.416+0000 7f2dd5d46640 4 set_mon_vals no callback set 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.416+0000 7f2dce7fc640 1 -- 172.21.15.52:0/1867743502 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2dc4001c40 con 0x7f2dd0142a20 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.416+0000 7f2dce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.416+0000 7f2dce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.416+0000 7f2dce7fc640 10 monclient: dump: 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.419 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.416+0000 7f2dd5d46640 10 monclient: shutdown 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.416+0000 7f2dd5d46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.416+0000 7f2dd5d46640 1 -- 172.21.15.52:0/1867743502 >> v1:172.21.15.179:6789/0 conn(0x7f2dd0142a20 legacy=0x7f2dd013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.417+0000 7f2dd5d46640 1 -- 172.21.15.52:0/1867743502 shutdown_connections 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.417+0000 7f2dd5d46640 1 -- 172.21.15.52:0/1867743502 >> 172.21.15.52:0/1867743502 conn(0x7f2dd00668e0 msgr2=0x7f2dd0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.417+0000 7f2dd5d46640 1 -- 172.21.15.52:0/1867743502 shutdown_connections 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.417+0000 7f2dd5d46640 1 -- 172.21.15.52:0/1867743502 wait complete. 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.418+0000 7f2dd5d46640 10 monclient: build_initial_monmap 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.418+0000 7f2dd5d46640 10 monclient: monmap: 2024-03-25T19:27:29.420 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 0 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 0.000000 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:created 0.000000 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 1 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.418+0000 7f2dd5d46640 1 librados: starting msgr at 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.418+0000 7f2dd5d46640 1 librados: starting objecter 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.418+0000 7f2dd5d46640 1 Processor -- start 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.419+0000 7f2dd5d46640 1 -- start start 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.419+0000 7f2dd5d46640 1 librados: setting wanted keys 2024-03-25T19:27:29.421 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.419+0000 7f2dd5d46640 1 librados: calling monclient init 2024-03-25T19:27:29.422 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.419+0000 7f2dd5d46640 10 monclient: init 2024-03-25T19:27:29.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.420+0000 7f2dd5d46640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:29.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.420+0000 7f2dd5d46640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:29.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.420+0000 7f2dd5d46640 10 monclient(hunting): picked mon.noname-c con 0x7f2dd00a96b0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:29.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.420+0000 7f2dd5d46640 10 monclient(hunting): picked mon.noname-a con 0x7f2dd00b4700 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:29.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.420+0000 7f2dd5d46640 10 monclient(hunting): picked mon.noname-b con 0x7f2dd0142a20 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:29.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.420+0000 7f2dd5d46640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd012aa20 con 0x7f2dd00b4700 2024-03-25T19:27:29.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.420+0000 7f2dd5d46640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd0141d80 con 0x7f2dd0142a20 2024-03-25T19:27:29.425 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.420+0000 7f2dd5d46640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd0142030 con 0x7f2dd00a96b0 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.420+0000 7f2dd5d46640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.420+0000 7f2dd5d46640 10 monclient(hunting): authenticate will time out at 5897.600789s 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.421+0000 7f2dcf7fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f2dd00a96b0 0x7f2dd0140110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:48076/0 (socket says 172.21.15.52:48076) 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.421+0000 7f2dcf7fe640 1 -- 172.21.15.52:0/156246095 learned_addr learned my addr 172.21.15.52:0/156246095 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.421+0000 7f2db7fff640 10 client.?.objecter ms_handle_connect 0x7f2dd00a96b0 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.421+0000 7f2db7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.421+0000 7f2db7fff640 10 client.?.objecter ms_handle_connect 0x7f2dd00b4700 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.421+0000 7f2db7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.422+0000 7f2db7fff640 10 client.?.objecter ms_handle_connect 0x7f2dd0142a20 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.422+0000 7f2db7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.424+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793567727 0 0) 0x7f2dd0142030 con 0x7f2dd00a96b0 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.424+0000 7f2db7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.424+0000 7f2db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.424+0000 7f2db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.424+0000 7f2db7fff640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.424+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dbc004eb0 con 0x7f2dd00a96b0 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927103361 0 0) 0x7f2dd012aa20 con 0x7f2dd00b4700 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 10 monclient(hunting): my global_id is 4167 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd0142030 con 0x7f2dd00b4700 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130556901 0 0) 0x7f2dd0141d80 con 0x7f2dd0142a20 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 10 monclient(hunting): my global_id is 4162 2024-03-25T19:27:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.425+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd012aa20 con 0x7f2dd0142a20 2024-03-25T19:27:29.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.432+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2803901407 0 0) 0x7f2dd0142030 con 0x7f2dd00b4700 2024-03-25T19:27:29.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.432+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dd0141d80 con 0x7f2dd00b4700 2024-03-25T19:27:29.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.432+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2dc4001b00 con 0x7f2dd00b4700 2024-03-25T19:27:29.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.432+0000 7f2db7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:29.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.432+0000 7f2db7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:29.433 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.432+0000 7f2db7fff640 10 monclient(hunting): dump: 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.432+0000 7f2db7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.432+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3660617988 0 0) 0x7f2dd012aa20 con 0x7f2dd0142a20 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.433+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dd0142030 con 0x7f2dd0142a20 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.433+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2dc8001c30 con 0x7f2dd0142a20 2024-03-25T19:27:29.434 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.433+0000 7f2db7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.433+0000 7f2db7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.433+0000 7f2db7fff640 10 monclient(hunting): dump: 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.433+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 983321725 0 0) 0x7f2dbc004eb0 con 0x7f2dd00a96b0 2024-03-25T19:27:29.435 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.433+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dd012aa20 con 0x7f2dd00a96b0 2024-03-25T19:27:29.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.434+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2dc00019f0 con 0x7f2dd00a96b0 2024-03-25T19:27:29.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.434+0000 7f2db7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:29.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.434+0000 7f2db7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:29.436 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.434+0000 7f2db7fff640 10 monclient(hunting): dump: 2024-03-25T19:27:29.436 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-25T19:27:29.437 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.437 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.437 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.437 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:29.437 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:29.437 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:29.437 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:29.437 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:29.437 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:29.437 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:29.434+0000 7f7f97048040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:27:29.439 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.438+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1604486021 0 0) 0x7f2dd0141d80 con 0x7f2dd00b4700 2024-03-25T19:27:29.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.438+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 >> v1:172.21.15.52:6790/0 conn(0x7f2dd00a96b0 legacy=0x7f2dd0140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.438+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 >> v1:172.21.15.179:6789/0 conn(0x7f2dd0142a20 legacy=0x7f2dd012a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.438+0000 7f2db7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:29.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.438+0000 7f2db7fff640 1 monclient: found mon.a 2024-03-25T19:27:29.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.438+0000 7f2db7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:29.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.438+0000 7f2db7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:29.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.438+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd01421c0 con 0x7f2dd00b4700 2024-03-25T19:27:29.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.438+0000 7f2db7fff640 10 monclient: _finish_auth 0 2024-03-25T19:27:29.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.438+0000 7f2db7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:29.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.438+0000 7f2db7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:29.440 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.439+0000 7f2dd5d46640 5 monclient: authenticate success, global_id 4167 2024-03-25T19:27:29.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.439+0000 7f2dd5d46640 10 monclient: _renew_subs 2024-03-25T19:27:29.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.439+0000 7f2dd5d46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:29.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.439+0000 7f2dd5d46640 1 -- 172.21.15.52:0/156246095 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd0131d60 con 0x7f2dd00b4700 2024-03-25T19:27:29.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.439+0000 7f2dd5d46640 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:29.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.439+0000 7f2dd5d46640 10 monclient: _renew_subs 2024-03-25T19:27:29.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.439+0000 7f2dd5d46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:29.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.439+0000 7f2dd5d46640 1 -- 172.21.15.52:0/156246095 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd012cac0 con 0x7f2dd00b4700 2024-03-25T19:27:29.441 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.439+0000 7f2dd5d46640 1 librados: init done 2024-03-25T19:27:29.442 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.441+0000 7f2db5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:29.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.441+0000 7f2db5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:29.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.441+0000 7f2db5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:29.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.441+0000 7f2db5ffb640 1 -- 172.21.15.52:0/156246095 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd0133c90 con 0x7f2dd00b4700 2024-03-25T19:27:29.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.441+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc4001cc0 con 0x7f2dd00b4700 2024-03-25T19:27:29.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.441+0000 7f2db7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:29.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.441+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2dc4003a40 con 0x7f2dd00b4700 2024-03-25T19:27:29.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.441+0000 7f2db7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:29.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.441+0000 7f2db7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:29.443 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.441+0000 7f2db7fff640 10 monclient: dump: 2024-03-25T19:27:29.443 INFO:teuthology.orchestra.run.smithi052.stderr:epoch 1 2024-03-25T19:27:29.443 INFO:teuthology.orchestra.run.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:29.444 INFO:teuthology.orchestra.run.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.444 INFO:teuthology.orchestra.run.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:29.444 INFO:teuthology.orchestra.run.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:29.444 INFO:teuthology.orchestra.run.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:29.444 INFO:teuthology.orchestra.run.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:29.444 INFO:teuthology.orchestra.run.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:29.444 INFO:teuthology.orchestra.run.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:29.444 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:27:29.444 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.441+0000 7f2dcdffb640 4 set_mon_vals no callback set 2024-03-25T19:27:29.482 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:29.482+0000 7f217b1db040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:27:29.494 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:29.495+0000 7f7f97048040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:27:29.538 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:29.539+0000 7f217b1db040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:27:29.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.615+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2dc4003f30 con 0x7f2dd00b4700 2024-03-25T19:27:29.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.615+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f2dc4005740 con 0x7f2dd00b4700 2024-03-25T19:27:29.615 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.615+0000 7f2db7fff640 10 client.4167.objecter ms_dispatch 0x7f2dd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:29.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.615+0000 7f2db7fff640 3 client.4167.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:29.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.615+0000 7f2db7fff640 3 client.4167.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:29.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.615+0000 7f2db7fff640 20 client.4167.objecter dump_active .. 0 homeless 2024-03-25T19:27:29.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.615+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2dc4005a80 con 0x7f2dd00b4700 2024-03-25T19:27:29.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.615+0000 7f2db7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:29.616 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.615+0000 7f2db7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:29.626 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:29.626+0000 7f7f97048040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:27:29.665 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-25T19:27:29.666+0000 7f217b1db040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:27:29.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.740+0000 7f2db5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:29.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.740+0000 7f2db5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:29.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.740+0000 7f2db5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:29.740 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.740+0000 7f2db5ffb640 1 -- 172.21.15.52:0/156246095 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dd00aab00 con 0x7f2dd00b4700 2024-03-25T19:27:29.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.825+0000 7f2db7fff640 1 -- 172.21.15.52:0/156246095 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f2dc4005120 con 0x7f2dd00b4700 2024-03-25T19:27:29.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.825+0000 7f2db7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:29.825 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.825+0000 7f2db7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:29.825 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-03-25T19:27:29.825 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:29.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 10 librados: watch_flush enter 2024-03-25T19:27:29.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 10 librados: watch_flush exit 2024-03-25T19:27:29.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 20 client.4167.objecter shutdown clearing up homeless session... 2024-03-25T19:27:29.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 10 client.4167.objecter successfully canceled tick 2024-03-25T19:27:29.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 10 monclient: shutdown 2024-03-25T19:27:29.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:29.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 1 -- 172.21.15.52:0/156246095 >> v1:172.21.15.52:6789/0 conn(0x7f2dd00b4700 legacy=0x7f2dd0129c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:29.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 1 -- 172.21.15.52:0/156246095 shutdown_connections 2024-03-25T19:27:29.827 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 1 -- 172.21.15.52:0/156246095 >> 172.21.15.52:0/156246095 conn(0x7f2dd00668e0 msgr2=0x7f2dd00b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:29.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 1 -- 172.21.15.52:0/156246095 shutdown_connections 2024-03-25T19:27:29.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 1 -- 172.21.15.52:0/156246095 wait complete. 2024-03-25T19:27:29.828 INFO:teuthology.orchestra.run.smithi052.stderr:2024-03-25T19:27:29.827+0000 7f2db5ffb640 1 librados: shutdown 2024-03-25T19:27:29.842 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:27:29.842 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:27:30.650 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:30.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.798+0000 7fab1a8e4640 10 monclient: get_monmap_and_config 2024-03-25T19:27:30.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.798+0000 7fab1a8e4640 10 monclient: build_initial_monmap 2024-03-25T19:27:30.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.798+0000 7fab1a8e4640 10 monclient: monmap: 2024-03-25T19:27:30.798 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:27:30.798 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:30.798 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:27:30.799077+0000 2024-03-25T19:27:30.798 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:27:30.799077+0000 2024-03-25T19:27:30.798 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:30.798 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:30.798 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:30.798 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:30.799 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:30.799 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:30.800 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.799+0000 7fab1a8e4640 1 Processor -- start 2024-03-25T19:27:30.800 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.799+0000 7fab1a8e4640 1 -- start start 2024-03-25T19:27:30.800 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.799+0000 7fab1a8e4640 10 monclient: init 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.800+0000 7fab1a8e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.800+0000 7fab1a8e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.800+0000 7fab1a8e4640 10 monclient(hunting): picked mon.noname-a con 0x7fab140a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.800+0000 7fab1a8e4640 10 monclient(hunting): picked mon.noname-c con 0x7fab1413f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.800+0000 7fab1a8e4640 10 monclient(hunting): picked mon.noname-b con 0x7fab141430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.800+0000 7fab1a8e4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab14142520 con 0x7fab140a98f0 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.800+0000 7fab1a8e4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab140b28a0 con 0x7fab1413f710 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.800+0000 7fab1a8e4640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab140b2a30 con 0x7fab141430c0 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.800+0000 7fab1a8e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.800+0000 7fab1a8e4640 10 monclient(hunting): authenticate will time out at 5898.980730s 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.801+0000 7fab13fff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fab140a98f0 0x7fab140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:55926/0 (socket says 172.21.15.52:55926) 2024-03-25T19:27:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.801+0000 7fab13fff640 1 -- 172.21.15.52:0/570901864 learned_addr learned my addr 172.21.15.52:0/570901864 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:30.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.804+0000 7fab12ffd640 1 -- 172.21.15.52:0/570901864 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906325893 0 0) 0x7fab14142520 con 0x7fab140a98f0 2024-03-25T19:27:30.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.805+0000 7fab12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:30.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.805+0000 7fab12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:30.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.805+0000 7fab12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:30.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.805+0000 7fab12ffd640 10 monclient(hunting): my global_id is 4170 2024-03-25T19:27:30.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.805+0000 7fab12ffd640 1 -- 172.21.15.52:0/570901864 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faafc003520 con 0x7fab140a98f0 2024-03-25T19:27:30.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 1 -- 172.21.15.52:0/570901864 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3141102519 0 0) 0x7faafc003520 con 0x7fab140a98f0 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 1 -- 172.21.15.52:0/570901864 >> v1:172.21.15.52:6790/0 conn(0x7fab141430c0 legacy=0x7fab1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 1 -- 172.21.15.52:0/570901864 >> v1:172.21.15.179:6789/0 conn(0x7fab1413f710 legacy=0x7fab141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 1 monclient: found mon.noname-a 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 1 -- 172.21.15.52:0/570901864 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab140b44c0 con 0x7fab140a98f0 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 10 monclient: _finish_auth 0 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 1 -- 172.21.15.52:0/570901864 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fab04001690 con 0x7fab140a98f0 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab12ffd640 10 monclient: dump: 2024-03-25T19:27:30.811 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:30.812 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:30.812 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:30.812 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:30.812 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:30.812 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:30.812 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:30.812 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:30.812 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:30.812 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:30.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab1a8e4640 5 monclient: authenticate success, global_id 4170 2024-03-25T19:27:30.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.810+0000 7fab1a8e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.813+0000 7fab12ffd640 1 -- 172.21.15.52:0/570901864 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab04001860 con 0x7fab140a98f0 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.813+0000 7fab12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.813+0000 7fab12ffd640 1 -- 172.21.15.52:0/570901864 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fab04001cb0 con 0x7fab140a98f0 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.813+0000 7fab1a8e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.813+0000 7fab1a8e4640 4 set_mon_vals no callback set 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.813+0000 7fab12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.813+0000 7fab12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.813+0000 7fab12ffd640 10 monclient: dump: 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:30.813 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:30.814 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:30.814 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:30.814 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:30.814 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:30.814 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:30.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.814+0000 7fab1a8e4640 10 monclient: shutdown 2024-03-25T19:27:30.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.814+0000 7fab1a8e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:30.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.814+0000 7fab1a8e4640 1 -- 172.21.15.52:0/570901864 >> v1:172.21.15.52:6789/0 conn(0x7fab140a98f0 legacy=0x7fab140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:30.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.814+0000 7fab1a8e4640 1 -- 172.21.15.52:0/570901864 shutdown_connections 2024-03-25T19:27:30.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.814+0000 7fab1a8e4640 1 -- 172.21.15.52:0/570901864 >> 172.21.15.52:0/570901864 conn(0x7fab140b7ea0 msgr2=0x7fab140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:30.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.815+0000 7fab1a8e4640 1 -- 172.21.15.52:0/570901864 shutdown_connections 2024-03-25T19:27:30.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.815+0000 7fab1a8e4640 1 -- 172.21.15.52:0/570901864 wait complete. 2024-03-25T19:27:30.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.816+0000 7fab1a8e4640 10 monclient: build_initial_monmap 2024-03-25T19:27:30.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.816+0000 7fab1a8e4640 10 monclient: monmap: 2024-03-25T19:27:30.816 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:27:30.816 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:30.816 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:27:30.816 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:27:30.816 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:30.816 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:27:30.816 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:30.816 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:30.816 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:30.817 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:30.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.816+0000 7fab1a8e4640 1 librados: starting msgr at 2024-03-25T19:27:30.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.816+0000 7fab1a8e4640 1 librados: starting objecter 2024-03-25T19:27:30.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.817+0000 7fab1a8e4640 1 Processor -- start 2024-03-25T19:27:30.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.817+0000 7fab1a8e4640 1 -- start start 2024-03-25T19:27:30.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.817+0000 7fab1a8e4640 1 librados: setting wanted keys 2024-03-25T19:27:30.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.817+0000 7fab1a8e4640 1 librados: calling monclient init 2024-03-25T19:27:30.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.817+0000 7fab1a8e4640 10 monclient: init 2024-03-25T19:27:30.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.818+0000 7fab1a8e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:30.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.818+0000 7fab1a8e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:30.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.819+0000 7fab1a8e4640 10 monclient(hunting): picked mon.noname-c con 0x7fab140a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:30.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.819+0000 7fab1a8e4640 10 monclient(hunting): picked mon.noname-a con 0x7fab1413f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:30.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.819+0000 7fab1a8e4640 10 monclient(hunting): picked mon.noname-b con 0x7fab141430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:30.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.819+0000 7fab1a8e4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab140af3f0 con 0x7fab1413f710 2024-03-25T19:27:30.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.819+0000 7fab1a8e4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab140b14f0 con 0x7fab141430c0 2024-03-25T19:27:30.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.819+0000 7fab1a8e4640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab140b1680 con 0x7fab140a98f0 2024-03-25T19:27:30.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.819+0000 7fab1a8e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:30.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.820+0000 7fab1a8e4640 10 monclient(hunting): authenticate will time out at 5898.999962s 2024-03-25T19:27:30.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.820+0000 7fab18e81640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fab141430c0 0x7fab1414d870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:58924/0 (socket says 172.21.15.52:58924) 2024-03-25T19:27:30.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.820+0000 7fab18e81640 1 -- 172.21.15.52:0/143013517 learned_addr learned my addr 172.21.15.52:0/143013517 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:30.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.820+0000 7fab10ff9640 10 client.?.objecter ms_handle_connect 0x7fab141430c0 2024-03-25T19:27:30.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.820+0000 7fab10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:30.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.821+0000 7fab10ff9640 10 client.?.objecter ms_handle_connect 0x7fab1413f710 2024-03-25T19:27:30.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.821+0000 7fab10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:30.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.825+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132613933 0 0) 0x7fab140af3f0 con 0x7fab1413f710 2024-03-25T19:27:30.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.825+0000 7fab10ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:30.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.825+0000 7fab10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:30.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.825+0000 7fab10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:30.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.825+0000 7fab10ff9640 10 monclient(hunting): my global_id is 4173 2024-03-25T19:27:30.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.825+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab00004eb0 con 0x7fab1413f710 2024-03-25T19:27:30.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.831+0000 7fab10ff9640 10 client.?.objecter ms_handle_connect 0x7fab140a98f0 2024-03-25T19:27:30.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.831+0000 7fab10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:30.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.836+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1412024380 0 0) 0x7fab00004eb0 con 0x7fab1413f710 2024-03-25T19:27:30.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.836+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab140af3f0 con 0x7fab1413f710 2024-03-25T19:27:30.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.836+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fab08001870 con 0x7fab1413f710 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.836+0000 7fab10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.836+0000 7fab10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.836+0000 7fab10ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:30.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.836+0000 7fab10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:30.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.966+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588130440 0 0) 0x7fab140b14f0 con 0x7fab141430c0 2024-03-25T19:27:30.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.966+0000 7fab10ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:30.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.966+0000 7fab10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:30.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.966+0000 7fab10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:30.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.966+0000 7fab10ff9640 10 monclient(hunting): my global_id is 4177 2024-03-25T19:27:30.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.966+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab00004eb0 con 0x7fab141430c0 2024-03-25T19:27:30.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.967+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358282893 0 0) 0x7fab140b1680 con 0x7fab140a98f0 2024-03-25T19:27:30.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.967+0000 7fab10ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:30.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.967+0000 7fab10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:30.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.967+0000 7fab10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:30.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.967+0000 7fab10ff9640 10 monclient(hunting): my global_id is 4169 2024-03-25T19:27:30.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.967+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab140b14f0 con 0x7fab140a98f0 2024-03-25T19:27:30.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.974+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3845314911 0 0) 0x7fab140b14f0 con 0x7fab140a98f0 2024-03-25T19:27:30.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.974+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab140b1680 con 0x7fab140a98f0 2024-03-25T19:27:30.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.975+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fab04001b70 con 0x7fab140a98f0 2024-03-25T19:27:30.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.975+0000 7fab10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:30.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.975+0000 7fab10ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:30.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.975+0000 7fab10ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:30.975 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:30.975 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:30.975 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:30.975 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:30.975 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:30.975 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:30.975 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:30.975 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:30.975 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:30.975 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:30.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.976+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3726387843 0 0) 0x7fab00004eb0 con 0x7fab141430c0 2024-03-25T19:27:30.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.976+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab140b14f0 con 0x7fab141430c0 2024-03-25T19:27:30.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.977+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fab0c004220 con 0x7fab141430c0 2024-03-25T19:27:30.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.977+0000 7fab10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:30.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.977+0000 7fab10ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:30.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.977+0000 7fab10ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:30.976 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:30.976 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:30.976 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:30.976 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:30.976 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:30.977 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:30.977 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:30.977 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:30.977 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:30.977 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:30.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1904614365 0 0) 0x7fab140af3f0 con 0x7fab1413f710 2024-03-25T19:27:30.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 >> v1:172.21.15.52:6790/0 conn(0x7fab140a98f0 legacy=0x7fab141d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:30.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 >> v1:172.21.15.179:6789/0 conn(0x7fab141430c0 legacy=0x7fab1414d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:30.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab10ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:27:30.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab10ff9640 1 monclient: found mon.a 2024-03-25T19:27:30.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:30.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab10ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:30.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1414df80 con 0x7fab1413f710 2024-03-25T19:27:30.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab10ff9640 10 monclient: _finish_auth 0 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab10ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab1a8e4640 5 monclient: authenticate success, global_id 4173 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab1a8e4640 10 monclient: _renew_subs 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab1a8e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab1a8e4640 1 -- 172.21.15.52:0/143013517 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1414e1d0 con 0x7fab1413f710 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab1a8e4640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab1a8e4640 10 monclient: _renew_subs 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab1a8e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab1a8e4640 1 -- 172.21.15.52:0/143013517 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1414e5e0 con 0x7fab1413f710 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.981+0000 7fab1a8e4640 1 librados: init done 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.982+0000 7fab1a8e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.982+0000 7fab1a8e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.982+0000 7fab1a8e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:30.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.983+0000 7fab1a8e4640 1 -- 172.21.15.52:0/143013517 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab140627b0 con 0x7fab1413f710 2024-03-25T19:27:30.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.988+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab080035f0 con 0x7fab1413f710 2024-03-25T19:27:30.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.988+0000 7fab10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:30.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:30.988+0000 7fab127fc640 4 set_mon_vals no callback set 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.029+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fab08003900 con 0x7fab1413f710 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.029+0000 7fab10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:31.028 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.029+0000 7fab10ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.029+0000 7fab10ff9640 10 monclient: dump: 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:31.029 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:31.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.033+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 461202723 0 0) 0x7fab0800e9e0 con 0x7fab1413f710 2024-03-25T19:27:31.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.064+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fab08014530 con 0x7fab1413f710 2024-03-25T19:27:31.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.064+0000 7fab10ff9640 10 client.4173.objecter ms_dispatch 0x7fab14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:31.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.064+0000 7fab10ff9640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:31.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.064+0000 7fab10ff9640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:31.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.064+0000 7fab10ff9640 20 client.4173.objecter dump_active .. 0 homeless 2024-03-25T19:27:31.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.186+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fab080139d0 con 0x7fab1413f710 2024-03-25T19:27:31.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.186+0000 7fab10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:31.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.186+0000 7fab10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:31.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.314+0000 7fab1a8e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:31.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.314+0000 7fab1a8e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:31.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.315+0000 7fab1a8e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:27:31.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.315+0000 7fab1a8e4640 1 -- 172.21.15.52:0/143013517 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab141421d0 con 0x7fab1413f710 2024-03-25T19:27:31.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.526+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 3354650494 0 0) 0x7fab08010470 con 0x7fab1413f710 2024-03-25T19:27:31.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.527+0000 7fab13fff640 10 monclient: get_auth_request con 0x7fab0006e4d0 auth_method 0 2024-03-25T19:27:31.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.527+0000 7fab10ff9640 10 client.4173.objecter ms_handle_connect 0x7fab0006e4d0 2024-03-25T19:27:31.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.575+0000 7fab10ff9640 1 -- 172.21.15.52:0/143013517 <== mon.0 v1:172.21.15.52:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fab0801fb50 con 0x7fab1413f710 2024-03-25T19:27:31.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.575+0000 7fab10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:31.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.575+0000 7fab10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:31.575 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:27:31.575 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 10 librados: watch_flush enter 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 10 librados: watch_flush exit 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 20 client.4173.objecter shutdown clearing up homeless session... 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 10 client.4173.objecter successfully canceled tick 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 1 -- 172.21.15.52:0/143013517 >> v1:172.21.15.52:6800/3967427147 conn(0x7fab0006e4d0 legacy=0x7fab000708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 10 monclient: shutdown 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 1 -- 172.21.15.52:0/143013517 >> v1:172.21.15.52:6789/0 conn(0x7fab1413f710 legacy=0x7fab140aece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 1 -- 172.21.15.52:0/143013517 shutdown_connections 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 1 -- 172.21.15.52:0/143013517 >> 172.21.15.52:0/143013517 conn(0x7fab140b7ea0 msgr2=0x7fab14074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 1 -- 172.21.15.52:0/143013517 shutdown_connections 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 1 -- 172.21.15.52:0/143013517 wait complete. 2024-03-25T19:27:31.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:31.576+0000 7fab1a8e4640 1 librados: shutdown 2024-03-25T19:27:31.594 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:31.847 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:31.847+0000 7f7f37cf4640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:27:31.891 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:27:31.892+0000 7f7f37cf4640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:27:36.165 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:06.837 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.195 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:06.843 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.196 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:06.872 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.236 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:06.909 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.254 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:06.904 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.281 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:06.957 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.378 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:07.050 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.435 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:07.107 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.645 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:07.304 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.723 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:07.383 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.740 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:07.395 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.767 INFO:tasks.ceph.osd.5.smithi179.stderr:2024-03-25T19:27:36.762+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:27:36.804 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:07.459 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.841 INFO:tasks.ceph.osd.7.smithi179.stderr:2024-03-25T19:27:36.834+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:27:36.844 INFO:tasks.ceph.osd.4.smithi179.stderr:2024-03-25T19:27:36.838+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:27:36.870 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:07.534 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.930 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:07.594 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:36.992 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:07.671 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.002 INFO:tasks.ceph.osd.6.smithi179.stderr:2024-03-25T19:27:36.996+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:27:37.053 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:07.732 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.239 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:07.911 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.251 INFO:tasks.ceph.osd.3.smithi052.stderr:2024-03-25T19:27:37.246+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:27:37.312 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:07.988 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.348 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:07.997 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.354 INFO:tasks.ceph.osd.2.smithi052.stderr:2024-03-25T19:27:37.349+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:27:37.358 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:08.030 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.466 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:08.138 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.511 INFO:tasks.ceph.osd.0.smithi052.stderr:2024-03-25T19:27:37.504+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:27:37.627 INFO:tasks.ceph.osd.1.smithi052.stderr:2024-03-25T19:27:37.621+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:27:37.678 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:08.350 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.681 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:08.357 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.713 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:08.362 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.723 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:08.383 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.821 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:08.475 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.831 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:08.503 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.947 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:08.619 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.975 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:08.639 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:37.993 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:08.669 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.041 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:08.690 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.106 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:08.765 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.106 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:08.786 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.150 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:08.822 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.187 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:08.841 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.211 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:08.883 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.224 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:08.903 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.257 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:08.933 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.312 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:08.961 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.341 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:09.004 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.400 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:38.414 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:09.086 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.432 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:09.090 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.475 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:09.147 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.489+0000 7f962c1cf640 10 monclient: get_monmap_and_config 2024-03-25T19:27:38.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.489+0000 7f962c1cf640 10 monclient: build_initial_monmap 2024-03-25T19:27:38.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.489+0000 7f962c1cf640 10 monclient: monmap: 2024-03-25T19:27:38.489 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:27:38.489 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:38.490 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:27:38.490220+0000 2024-03-25T19:27:38.490 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:27:38.490220+0000 2024-03-25T19:27:38.490 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:38.490 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:38.490 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:38.490 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:38.490 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:38.490 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.490+0000 7f962c1cf640 1 Processor -- start 2024-03-25T19:27:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.490+0000 7f962c1cf640 1 -- start start 2024-03-25T19:27:38.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.490+0000 7f962c1cf640 10 monclient: init 2024-03-25T19:27:38.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.490+0000 7f962c1cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:38.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.490+0000 7f962c1cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:38.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.490+0000 7f962c1cf640 10 monclient(hunting): picked mon.noname-b con 0x7f96240ae6e0 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.490+0000 7f962c1cf640 10 monclient(hunting): picked mon.noname-a con 0x7f962406f080 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.490+0000 7f962c1cf640 10 monclient(hunting): picked mon.noname-c con 0x7f96240704e0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.490+0000 7f962c1cf640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96240718d0 con 0x7f962406f080 2024-03-25T19:27:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.490+0000 7f962c1cf640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962406fa30 con 0x7f96240704e0 2024-03-25T19:27:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.491+0000 7f962c1cf640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9624070090 con 0x7f96240ae6e0 2024-03-25T19:27:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.491+0000 7f962c1cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.491+0000 7f962c1cf640 10 monclient(hunting): authenticate will time out at 5906.670928s 2024-03-25T19:27:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.491+0000 7f962976a640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f962406f080 0x7f9624071c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:35004/0 (socket says 172.21.15.52:35004) 2024-03-25T19:27:38.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.491+0000 7f962976a640 1 -- 172.21.15.52:0/3048380644 learned_addr learned my addr 172.21.15.52:0/3048380644 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:38.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.493+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557844510 0 0) 0x7f96240718d0 con 0x7f962406f080 2024-03-25T19:27:38.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.493+0000 7f9628f69640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.493+0000 7f9628f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.493+0000 7f9628f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.493+0000 7f9628f69640 10 monclient(hunting): my global_id is 4200 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.493+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9610005730 con 0x7f962406f080 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.493+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779903145 0 0) 0x7f9624070090 con 0x7f96240ae6e0 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.493+0000 7f9628f69640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.493+0000 7f9628f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.493+0000 7f9628f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.494+0000 7f9628f69640 10 monclient(hunting): my global_id is 4196 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.494+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96240718d0 con 0x7f96240ae6e0 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.494+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242572863 0 0) 0x7f962406fa30 con 0x7f96240704e0 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.494+0000 7f9628f69640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.494+0000 7f9628f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.494+0000 7f9628f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.494+0000 7f9628f69640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:27:38.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.494+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9624070090 con 0x7f96240704e0 2024-03-25T19:27:38.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 475465450 0 0) 0x7f96240718d0 con 0x7f96240ae6e0 2024-03-25T19:27:38.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 >> v1:172.21.15.179:6789/0 conn(0x7f96240704e0 legacy=0x7f962414a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 >> v1:172.21.15.52:6789/0 conn(0x7f962406f080 legacy=0x7f9624071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f9628f69640 10 monclient: _finish_hunting 0 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f9628f69640 1 monclient: found mon.noname-b 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f9628f69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f9628f69640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962406f510 con 0x7f96240ae6e0 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f9628f69640 10 monclient: _finish_auth 0 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f9628f69640 10 monclient: _check_auth_tickets 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f9628f69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f962c1cf640 5 monclient: authenticate success, global_id 4196 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.496+0000 7f962c1cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.497+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9620001630 con 0x7f96240ae6e0 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.497+0000 7f9628f69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.497+0000 7f9628f69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.497+0000 7f9628f69640 10 monclient: dump: 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:38.497 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.498 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.498 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:38.498 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:38.498 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:38.498 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:38.498 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:38.498 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:38.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.497+0000 7f962c1cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:38.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9620001880 con 0x7f96240ae6e0 2024-03-25T19:27:38.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f9628f69640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:38.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f9628f69640 1 -- 172.21.15.52:0/3048380644 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9620001c10 con 0x7f96240ae6e0 2024-03-25T19:27:38.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f962c1cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:38.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f9628f69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:38.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f962c1cf640 4 set_mon_vals no callback set 2024-03-25T19:27:38.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f9628f69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:38.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f9628f69640 10 monclient: dump: 2024-03-25T19:27:38.500 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:38.500 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:38.500 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f962c1cf640 10 monclient: shutdown 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f962c1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f962c1cf640 1 -- 172.21.15.52:0/3048380644 >> v1:172.21.15.52:6790/0 conn(0x7f96240ae6e0 legacy=0x7f962413c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f962c1cf640 1 -- 172.21.15.52:0/3048380644 shutdown_connections 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f962c1cf640 1 -- 172.21.15.52:0/3048380644 >> 172.21.15.52:0/3048380644 conn(0x7f962406b630 msgr2=0x7f96240b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f962c1cf640 1 -- 172.21.15.52:0/3048380644 shutdown_connections 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.499+0000 7f962c1cf640 1 -- 172.21.15.52:0/3048380644 wait complete. 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.500+0000 7f962c1cf640 10 monclient: build_initial_monmap 2024-03-25T19:27:38.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.500+0000 7f962c1cf640 10 monclient: monmap: 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.500+0000 7f962c1cf640 1 librados: starting msgr at 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.500+0000 7f962c1cf640 1 librados: starting objecter 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.500+0000 7f962c1cf640 1 Processor -- start 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.500+0000 7f962c1cf640 1 -- start start 2024-03-25T19:27:38.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.500+0000 7f962c1cf640 1 librados: setting wanted keys 2024-03-25T19:27:38.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.500+0000 7f962c1cf640 1 librados: calling monclient init 2024-03-25T19:27:38.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.500+0000 7f962c1cf640 10 monclient: init 2024-03-25T19:27:38.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f962c1cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:38.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f962c1cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:38.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f962c1cf640 10 monclient(hunting): picked mon.noname-a con 0x7f96240704e0 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:38.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f962c1cf640 10 monclient(hunting): picked mon.noname-b con 0x7f96240ae6e0 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f962c1cf640 10 monclient(hunting): picked mon.noname-c con 0x7f962413f470 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f962c1cf640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9624137620 con 0x7f96240704e0 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f962c1cf640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962413e860 con 0x7f96240ae6e0 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f962c1cf640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962413cdf0 con 0x7f962413f470 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f962c1cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f962c1cf640 10 monclient(hunting): authenticate will time out at 5906.681427s 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f9629f6b640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f96240704e0 0x7f96241daaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:35016/0 (socket says 172.21.15.52:35016) 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.501+0000 7f9629f6b640 1 -- 172.21.15.52:0/748705263 learned_addr learned my addr 172.21.15.52:0/748705263 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.502+0000 7f961a7fc640 10 client.?.objecter ms_handle_connect 0x7f96240704e0 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.502+0000 7f961a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.502+0000 7f961a7fc640 10 client.?.objecter ms_handle_connect 0x7f962413f470 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.502+0000 7f961a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.502+0000 7f961a7fc640 10 client.?.objecter ms_handle_connect 0x7f96240ae6e0 2024-03-25T19:27:38.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.502+0000 7f961a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.503+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062044923 0 0) 0x7f9624137620 con 0x7f96240704e0 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.503+0000 7f961a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.503+0000 7f961a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.503+0000 7f961a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.503+0000 7f961a7fc640 10 monclient(hunting): my global_id is 4203 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.503+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9608004eb0 con 0x7f96240704e0 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189405616 0 0) 0x7f962413e860 con 0x7f96240ae6e0 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9624137620 con 0x7f96240ae6e0 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940756781 0 0) 0x7f962413cdf0 con 0x7f962413f470 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:38.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:38.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 10 monclient(hunting): my global_id is 4199 2024-03-25T19:27:38.507 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.504+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f962413e860 con 0x7f962413f470 2024-03-25T19:27:38.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2831281228 0 0) 0x7f9624137620 con 0x7f96240ae6e0 2024-03-25T19:27:38.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f962413cdf0 con 0x7f96240ae6e0 2024-03-25T19:27:38.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9614001950 con 0x7f96240ae6e0 2024-03-25T19:27:38.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:38.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:38.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 10 monclient(hunting): dump: 2024-03-25T19:27:38.508 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:38.508 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:38.508 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.508 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4258648054 0 0) 0x7f9608004eb0 con 0x7f96240704e0 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9624137620 con 0x7f96240704e0 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9620001c50 con 0x7f96240704e0 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 10 monclient(hunting): dump: 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.509 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.508+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448110293 0 0) 0x7f962413e860 con 0x7f962413f470 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.509+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9608004eb0 con 0x7f962413f470 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.509+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f961c001bd0 con 0x7f962413f470 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.509+0000 7f961a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.509+0000 7f961a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.509+0000 7f961a7fc640 10 monclient(hunting): dump: 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:38.510 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.511 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.511 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:38.511 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:38.511 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:38.511 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:38.511 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:38.511 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:38.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.511+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2359849793 0 0) 0x7f962413cdf0 con 0x7f96240ae6e0 2024-03-25T19:27:38.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.511+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 >> v1:172.21.15.52:6790/0 conn(0x7f962413f470 legacy=0x7f9624136f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.511+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 >> v1:172.21.15.52:6789/0 conn(0x7f96240704e0 legacy=0x7f96241daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:38.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.511+0000 7f961a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:38.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.511+0000 7f961a7fc640 1 monclient: found mon.b 2024-03-25T19:27:38.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.511+0000 7f961a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:38.512 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.511+0000 7f961a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.511+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962413e430 con 0x7f96240ae6e0 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.511+0000 7f961a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.511+0000 7f961a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.511+0000 7f961a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.512+0000 7f962c1cf640 5 monclient: authenticate success, global_id 4207 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.512+0000 7f962c1cf640 10 monclient: _renew_subs 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.512+0000 7f962c1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.512+0000 7f962c1cf640 1 -- 172.21.15.52:0/748705263 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96241c6370 con 0x7f96240ae6e0 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.512+0000 7f962c1cf640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.512+0000 7f962c1cf640 10 monclient: _renew_subs 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.512+0000 7f962c1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.512+0000 7f962c1cf640 1 -- 172.21.15.52:0/748705263 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f962413ef60 con 0x7f96240ae6e0 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.512+0000 7f962c1cf640 1 librados: init done 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.513+0000 7f962c1cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.513+0000 7f962c1cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.513+0000 7f962c1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:38.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.513+0000 7f962c1cf640 1 -- 172.21.15.52:0/748705263 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962413de80 con 0x7f96240ae6e0 2024-03-25T19:27:38.522 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:09.198 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.529+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9614001d60 con 0x7f96240ae6e0 2024-03-25T19:27:38.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.529+0000 7f961a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.529+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9614004980 con 0x7f96240ae6e0 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.529+0000 7f961a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.529+0000 7f961a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.529+0000 7f961a7fc640 10 monclient: dump: 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:38.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.529+0000 7f961bfff640 4 set_mon_vals no callback set 2024-03-25T19:27:38.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.530+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f961401a950 con 0x7f96240ae6e0 2024-03-25T19:27:38.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.531+0000 7f9629f6b640 10 monclient: get_auth_request con 0x7f960806e060 auth_method 0 2024-03-25T19:27:38.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.531+0000 7f961a7fc640 10 client.4207.objecter ms_handle_connect 0x7f960806e060 2024-03-25T19:27:38.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.531+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f96140546d0 con 0x7f96240ae6e0 2024-03-25T19:27:38.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.531+0000 7f961a7fc640 10 client.4207.objecter ms_dispatch 0x7f9624000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:38.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.531+0000 7f961a7fc640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:38.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.531+0000 7f961a7fc640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:38.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.531+0000 7f961a7fc640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T19:27:38.555 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:09.234 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.556 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:09.210 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.577 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:09.226 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.654 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:09.318 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.681 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:09.353 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.695 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:09.355 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.739 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:09.411 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.760+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9614050010 con 0x7f96240ae6e0 2024-03-25T19:27:38.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.760+0000 7f961a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:38.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.760+0000 7f961a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:38.791 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:09.467 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.822 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:09.502 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.842 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:09.491 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.851 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:09.505 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.922 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:09.586 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.945 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:09.617 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:38.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.963+0000 7f962c1cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:38.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.963+0000 7f962c1cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:38.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.963+0000 7f962c1cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:38.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:38.963+0000 7f962c1cf640 1 -- 172.21.15.52:0/748705263 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96241c6710 con 0x7f96240ae6e0 2024-03-25T19:27:38.963 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:09.623 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.003 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:09.675 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.009 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.010+0000 7f961a7fc640 1 -- 172.21.15.52:0/748705263 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f9614052110 con 0x7f96240ae6e0 2024-03-25T19:27:39.009 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.010+0000 7f961a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:39.009 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.010+0000 7f961a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:39.009 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:27:39.010 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:39.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 10 librados: watch_flush enter 2024-03-25T19:27:39.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 10 librados: watch_flush exit 2024-03-25T19:27:39.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T19:27:39.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 10 client.4207.objecter successfully canceled tick 2024-03-25T19:27:39.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 1 -- 172.21.15.52:0/748705263 >> v1:172.21.15.52:6800/3967427147 conn(0x7f960806e060 legacy=0x7f9608070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:39.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 10 monclient: shutdown 2024-03-25T19:27:39.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:39.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 1 -- 172.21.15.52:0/748705263 >> v1:172.21.15.179:6789/0 conn(0x7f96240ae6e0 legacy=0x7f9624136800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:39.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 1 -- 172.21.15.52:0/748705263 shutdown_connections 2024-03-25T19:27:39.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 1 -- 172.21.15.52:0/748705263 >> 172.21.15.52:0/748705263 conn(0x7f962406b630 msgr2=0x7f962413c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:39.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 1 -- 172.21.15.52:0/748705263 shutdown_connections 2024-03-25T19:27:39.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 1 -- 172.21.15.52:0/748705263 wait complete. 2024-03-25T19:27:39.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:39.012+0000 7f962c1cf640 1 librados: shutdown 2024-03-25T19:27:39.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:39.055 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:09.731 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.094 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:09.774 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.114 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:09.763 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.158 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:09.813 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.196 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:09.860 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.213 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:09.885 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.242 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:09.901 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.267 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:09.939 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.323 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:09.999 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.359 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:10.038 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.378 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:10.027 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.447 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:10.102 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.477 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:10.149 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.477 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:10.141 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.507 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:10.167 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.590 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:10.266 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.626 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:10.306 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.643 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:10.292 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.731 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:10.403 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.738 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:10.393 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.748 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:10.420 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.750 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:10.413 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.774 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:10.434 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.785 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:10.457 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:39.891 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:10.570 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.033 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:10.696 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.037 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:10.692 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.044 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:10.704 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.093 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:10.769 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.147 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:10.823 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.158 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:10.807 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.160 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:10.839 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.212 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:10.862 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.259 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:10.931 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.297 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:10.960 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.313 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:10.986 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.333 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:10.987 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.534 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:11.193 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.589 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:11.249 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.638 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:11.317 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.698 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:11.377 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.782 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:11.446 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.869 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:11.532 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.874 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:11.528 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:40.930 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:11.584 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:45.022 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:15.694 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:45.444 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:16.120 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:45.749 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:16.421 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:45.781 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:16.428 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:45.827 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:45.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.964+0000 7f5618ba3640 10 monclient: get_monmap_and_config 2024-03-25T19:27:45.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.964+0000 7f5618ba3640 10 monclient: build_initial_monmap 2024-03-25T19:27:45.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.964+0000 7f5618ba3640 10 monclient: monmap: 2024-03-25T19:27:45.964 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:27:45.965 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:45.965 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:27:45.965486+0000 2024-03-25T19:27:45.965 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:27:45.965486+0000 2024-03-25T19:27:45.965 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:45.965 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:45.965 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:45.965 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:45.965 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:45.965 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:45.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.965+0000 7f5618ba3640 1 Processor -- start 2024-03-25T19:27:45.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.966+0000 7f5618ba3640 1 -- start start 2024-03-25T19:27:45.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.966+0000 7f5618ba3640 10 monclient: init 2024-03-25T19:27:45.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.966+0000 7f5618ba3640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:45.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.966+0000 7f5618ba3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:45.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.966+0000 7f5618ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f56140b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:45.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.966+0000 7f5618ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f56140b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:45.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.967+0000 7f5618ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f561414be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:45.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.967+0000 7f5618ba3640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56140af940 con 0x7f56140b4d60 2024-03-25T19:27:45.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.967+0000 7f5618ba3640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56140b0a70 con 0x7f56140b1730 2024-03-25T19:27:45.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.967+0000 7f5618ba3640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56140b0d50 con 0x7f561414be90 2024-03-25T19:27:45.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.967+0000 7f5618ba3640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:45.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.967+0000 7f5618ba3640 10 monclient(hunting): authenticate will time out at 5914.147014s 2024-03-25T19:27:45.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.967+0000 7f561259c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f56140b4d60 0x7f561413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:46732/0 (socket says 172.21.15.52:46732) 2024-03-25T19:27:45.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.967+0000 7f561259c640 1 -- 172.21.15.52:0/871143530 learned_addr learned my addr 172.21.15.52:0/871143530 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:45.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276020038 0 0) 0x7f56140b0a70 con 0x7f56140b1730 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5600003520 con 0x7f56140b1730 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321323461 0 0) 0x7f56140af940 con 0x7f56140b4d60 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56140b0a70 con 0x7f56140b4d60 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470042687 0 0) 0x7f56140b0d50 con 0x7f561414be90 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:27:45.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:45.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.971+0000 7f5611d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:45.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.972+0000 7f5611d9b640 10 monclient(hunting): my global_id is 4202 2024-03-25T19:27:45.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.972+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56140af940 con 0x7f561414be90 2024-03-25T19:27:45.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2364558309 0 0) 0x7f5600003520 con 0x7f56140b1730 2024-03-25T19:27:45.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 >> v1:172.21.15.52:6790/0 conn(0x7f561414be90 legacy=0x7f561414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:45.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 >> v1:172.21.15.52:6789/0 conn(0x7f56140b4d60 legacy=0x7f561413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:45.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 1 monclient: found mon.noname-c 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56140b1030 con 0x7f56140b1730 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 10 monclient: _finish_auth 0 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f55fc001690 con 0x7f56140b1730 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5618ba3640 5 monclient: authenticate success, global_id 4210 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5618ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5611d9b640 10 monclient: dump: 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:45.976 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:45.977 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:45.977 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:45.977 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:45.977 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:45.977 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:45.977 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:45.977 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:45.977 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:45.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.976+0000 7f5618ba3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:45.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.978+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55fc0018b0 con 0x7f56140b1730 2024-03-25T19:27:45.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.978+0000 7f5611d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:45.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.978+0000 7f5611d9b640 1 -- 172.21.15.52:0/871143530 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f55fc001c40 con 0x7f56140b1730 2024-03-25T19:27:45.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.978+0000 7f5611d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:45.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.978+0000 7f5611d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:45.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.978+0000 7f5611d9b640 10 monclient: dump: 2024-03-25T19:27:45.979 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:45.979 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:45.979 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:45.979 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:45.979 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.978+0000 7f5618ba3640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.978+0000 7f5618ba3640 4 set_mon_vals no callback set 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.979+0000 7f5618ba3640 10 monclient: shutdown 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.979+0000 7f5618ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.979+0000 7f5618ba3640 1 -- 172.21.15.52:0/871143530 >> v1:172.21.15.179:6789/0 conn(0x7f56140b1730 legacy=0x7f56140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.979+0000 7f5618ba3640 1 -- 172.21.15.52:0/871143530 shutdown_connections 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.979+0000 7f5618ba3640 1 -- 172.21.15.52:0/871143530 >> 172.21.15.52:0/871143530 conn(0x7f56140617a0 msgr2=0x7f5614061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.979+0000 7f5618ba3640 1 -- 172.21.15.52:0/871143530 shutdown_connections 2024-03-25T19:27:45.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.979+0000 7f5618ba3640 1 -- 172.21.15.52:0/871143530 wait complete. 2024-03-25T19:27:45.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.980+0000 7f5618ba3640 10 monclient: build_initial_monmap 2024-03-25T19:27:45.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.980+0000 7f5618ba3640 10 monclient: monmap: 2024-03-25T19:27:45.983 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:27:45.983 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:45.983 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.980+0000 7f5618ba3640 1 librados: starting msgr at 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.980+0000 7f5618ba3640 1 librados: starting objecter 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.980+0000 7f5618ba3640 1 Processor -- start 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.981+0000 7f5618ba3640 1 -- start start 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.981+0000 7f5618ba3640 1 librados: setting wanted keys 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.981+0000 7f5618ba3640 1 librados: calling monclient init 2024-03-25T19:27:45.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.981+0000 7f5618ba3640 10 monclient: init 2024-03-25T19:27:45.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.981+0000 7f5618ba3640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:45.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.981+0000 7f5618ba3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:45.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.981+0000 7f5618ba3640 10 monclient(hunting): picked mon.noname-b con 0x7f56140b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:45.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.982+0000 7f5618ba3640 10 monclient(hunting): picked mon.noname-c con 0x7f56140b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.982+0000 7f5618ba3640 10 monclient(hunting): picked mon.noname-a con 0x7f561414be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.982+0000 7f5618ba3640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5614145540 con 0x7f561414be90 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.982+0000 7f5618ba3640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561413ac50 con 0x7f56140b1730 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.982+0000 7f5618ba3640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561412f7b0 con 0x7f56140b4d60 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.982+0000 7f5618ba3640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.982+0000 7f5618ba3640 10 monclient(hunting): authenticate will time out at 5914.162208s 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.982+0000 7f561259c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f56140b1730 0x7f5614139190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:52072/0 (socket says 172.21.15.52:52072) 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.982+0000 7f561259c640 1 -- 172.21.15.52:0/4001450724 learned_addr learned my addr 172.21.15.52:0/4001450724 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.983+0000 7f560affd640 10 client.?.objecter ms_handle_connect 0x7f56140b1730 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.983+0000 7f560affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.983+0000 7f560affd640 10 client.?.objecter ms_handle_connect 0x7f561414be90 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.983+0000 7f560affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:45.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.983+0000 7f560affd640 10 client.?.objecter ms_handle_connect 0x7f56140b4d60 2024-03-25T19:27:45.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.983+0000 7f560affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.985+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935978266 0 0) 0x7f561413ac50 con 0x7f56140b1730 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.985+0000 7f560affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.985+0000 7f560affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.985+0000 7f560affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.985+0000 7f560affd640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.985+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f4004eb0 con 0x7f56140b1730 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.986+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525053289 0 0) 0x7f561412f7b0 con 0x7f56140b4d60 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.986+0000 7f560affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.986+0000 7f560affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.986+0000 7f560affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.986+0000 7f560affd640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.986+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f561413ac50 con 0x7f56140b4d60 2024-03-25T19:27:45.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.987+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065961302 0 0) 0x7f5614145540 con 0x7f561414be90 2024-03-25T19:27:45.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.987+0000 7f560affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:45.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.987+0000 7f560affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:45.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.987+0000 7f560affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:45.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.987+0000 7f560affd640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:27:45.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.987+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f561412f7b0 con 0x7f561414be90 2024-03-25T19:27:45.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.993+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 77904967 0 0) 0x7f55f4004eb0 con 0x7f56140b1730 2024-03-25T19:27:45.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.993+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5614145540 con 0x7f56140b1730 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.993+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f56040019f0 con 0x7f56140b1730 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.993+0000 7f560affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.993+0000 7f560affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.993+0000 7f560affd640 10 monclient(hunting): dump: 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:45.993 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:45.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.993+0000 7f560affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:45.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.994+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3105178096 0 0) 0x7f561413ac50 con 0x7f56140b4d60 2024-03-25T19:27:45.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.994+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55f4004eb0 con 0x7f56140b4d60 2024-03-25T19:27:45.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.994+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f55fc0015d0 con 0x7f56140b4d60 2024-03-25T19:27:45.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.994+0000 7f560affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:45.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.994+0000 7f560affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:45.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.994+0000 7f560affd640 10 monclient(hunting): dump: 2024-03-25T19:27:45.995 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.994+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3792908797 0 0) 0x7f561412f7b0 con 0x7f561414be90 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.994+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f561413ac50 con 0x7f561414be90 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.995+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f560c001b10 con 0x7f561414be90 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.995+0000 7f560affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.995+0000 7f560affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:45.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.995+0000 7f560affd640 10 monclient(hunting): dump: 2024-03-25T19:27:45.997 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:45.997 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:45.997 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:45.997 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:45.997 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:45.997 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:45.997 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:45.997 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:45.997 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:45.997 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:46.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3741037547 0 0) 0x7f5614145540 con 0x7f56140b1730 2024-03-25T19:27:46.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 >> v1:172.21.15.52:6790/0 conn(0x7f56140b4d60 legacy=0x7f561412e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:46.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 >> v1:172.21.15.52:6789/0 conn(0x7f561414be90 legacy=0x7f561412f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:46.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f560affd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:46.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f560affd640 1 monclient: found mon.b 2024-03-25T19:27:46.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f560affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:46.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f560affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561413af30 con 0x7f56140b1730 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f560affd640 10 monclient: _finish_auth 0 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f560affd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f560affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f5618ba3640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f5618ba3640 10 monclient: _renew_subs 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f5618ba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f5618ba3640 1 -- 172.21.15.52:0/4001450724 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56141451d0 con 0x7f56140b1730 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f5618ba3640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f5618ba3640 10 monclient: _renew_subs 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f5618ba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f5618ba3640 1 -- 172.21.15.52:0/4001450724 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5614139ca0 con 0x7f56140b1730 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:45.998+0000 7f5618ba3640 1 librados: init done 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.000+0000 7f5618ba3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:46.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.000+0000 7f5618ba3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:46.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.000+0000 7f5618ba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:46.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.000+0000 7f5618ba3640 1 -- 172.21.15.52:0/4001450724 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56141401d0 con 0x7f56140b1730 2024-03-25T19:27:46.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.021+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5604001b50 con 0x7f56140b1730 2024-03-25T19:27:46.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.021+0000 7f560affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:46.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.021+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f5604001d70 con 0x7f56140b1730 2024-03-25T19:27:46.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.021+0000 7f560affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:46.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.021+0000 7f560affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:46.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.021+0000 7f560affd640 10 monclient: dump: 2024-03-25T19:27:46.021 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:46.021 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:46.021 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:46.021 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:46.021 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:46.022 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:46.022 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:46.022 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:46.022 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:46.022 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:46.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.021+0000 7f561159a640 4 set_mon_vals no callback set 2024-03-25T19:27:46.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.022+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f560401ab60 con 0x7f56140b1730 2024-03-25T19:27:46.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.023+0000 7f560bfff640 10 monclient: get_auth_request con 0x7f55f406e0f0 auth_method 0 2024-03-25T19:27:46.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.023+0000 7f560affd640 10 client.4213.objecter ms_handle_connect 0x7f55f406e0f0 2024-03-25T19:27:46.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.025+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f56040548b0 con 0x7f56140b1730 2024-03-25T19:27:46.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.025+0000 7f560affd640 10 client.4213.objecter ms_dispatch 0x7f5614000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:46.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.025+0000 7f560affd640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:46.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.025+0000 7f560affd640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:46.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.025+0000 7f560affd640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-25T19:27:46.209 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:16.868 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:46.238 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:16.917 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:46.290 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:16.945 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:46.353 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:17.017 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:46.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.360+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56040501f0 con 0x7f56140b1730 2024-03-25T19:27:46.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.360+0000 7f560affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:46.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.360+0000 7f560affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:46.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.625+0000 7f5618ba3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:46.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.626+0000 7f5618ba3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:46.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.626+0000 7f5618ba3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:27:46.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.626+0000 7f5618ba3640 1 -- 172.21.15.52:0/4001450724 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56140627b0 con 0x7f56140b1730 2024-03-25T19:27:46.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.643+0000 7f560affd640 1 -- 172.21.15.52:0/4001450724 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f56040522f0 con 0x7f56140b1730 2024-03-25T19:27:46.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.643+0000 7f560affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:46.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.643+0000 7f560affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:46.643 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:27:46.643 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:46.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.645+0000 7f5618ba3640 10 librados: watch_flush enter 2024-03-25T19:27:46.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.645+0000 7f5618ba3640 10 librados: watch_flush exit 2024-03-25T19:27:46.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.645+0000 7f5618ba3640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-25T19:27:46.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.645+0000 7f5618ba3640 10 client.4213.objecter successfully canceled tick 2024-03-25T19:27:46.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.646+0000 7f5618ba3640 1 -- 172.21.15.52:0/4001450724 >> v1:172.21.15.52:6800/3967427147 conn(0x7f55f406e0f0 legacy=0x7f55f40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:46.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.646+0000 7f5618ba3640 10 monclient: shutdown 2024-03-25T19:27:46.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.646+0000 7f5618ba3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:46.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.646+0000 7f5618ba3640 1 -- 172.21.15.52:0/4001450724 >> v1:172.21.15.179:6789/0 conn(0x7f56140b1730 legacy=0x7f5614139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:46.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.646+0000 7f5618ba3640 1 -- 172.21.15.52:0/4001450724 shutdown_connections 2024-03-25T19:27:46.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.646+0000 7f5618ba3640 1 -- 172.21.15.52:0/4001450724 >> 172.21.15.52:0/4001450724 conn(0x7f56140617a0 msgr2=0x7f56140b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:46.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.646+0000 7f5618ba3640 1 -- 172.21.15.52:0/4001450724 shutdown_connections 2024-03-25T19:27:46.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.646+0000 7f5618ba3640 1 -- 172.21.15.52:0/4001450724 wait complete. 2024-03-25T19:27:46.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:46.646+0000 7f5618ba3640 1 librados: shutdown 2024-03-25T19:27:46.656 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:50.642 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:21.314 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:50.646 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:21.318 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.100 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:21.776 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.104 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:21.780 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.220 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:21.869 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.224 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:21.873 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.508 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:22.180 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.514 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:22.186 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.805 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:22.484 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.809 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:22.488 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.814 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:22.474 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.818 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:22.478 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.847 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:22.511 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.851 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:22.515 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.855 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:22.510 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:51.860 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:22.514 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:53.463 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:53.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.556+0000 7f24de40f640 10 monclient: get_monmap_and_config 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.556+0000 7f24de40f640 10 monclient: build_initial_monmap 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.556+0000 7f24de40f640 10 monclient: monmap: 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:27:53.556859+0000 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:27:53.556859+0000 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:27:53.556 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:53.557 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.557+0000 7f24de40f640 1 Processor -- start 2024-03-25T19:27:53.557 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.557+0000 7f24de40f640 1 -- start start 2024-03-25T19:27:53.557 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.557+0000 7f24de40f640 10 monclient: init 2024-03-25T19:27:53.557 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24de40f640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:53.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24de40f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:53.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24de40f640 10 monclient(hunting): picked mon.noname-b con 0x7f24d80b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:53.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24de40f640 10 monclient(hunting): picked mon.noname-a con 0x7f24d80b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:53.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24de40f640 10 monclient(hunting): picked mon.noname-c con 0x7f24d814be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:53.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24de40f640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d80af940 con 0x7f24d80b1730 2024-03-25T19:27:53.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24d77fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f24d80b1730 0x7f24d80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:46748/0 (socket says 172.21.15.52:46748) 2024-03-25T19:27:53.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24d77fe640 1 -- 172.21.15.52:0/2080767325 learned_addr learned my addr 172.21.15.52:0/2080767325 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:53.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24de40f640 1 -- 172.21.15.52:0/2080767325 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d80b0a70 con 0x7f24d814be90 2024-03-25T19:27:53.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24de40f640 1 -- 172.21.15.52:0/2080767325 --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d80b0d50 con 0x7f24d80b4d60 2024-03-25T19:27:53.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24de40f640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:53.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.558+0000 7f24de40f640 10 monclient(hunting): authenticate will time out at 5921.738798s 2024-03-25T19:27:53.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.562+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140752593 0 0) 0x7f24d80b0a70 con 0x7f24d814be90 2024-03-25T19:27:53.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.562+0000 7f24d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:53.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.562+0000 7f24d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:53.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.562+0000 7f24d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:53.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.562+0000 7f24d6ffd640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:27:53.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.562+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c0003520 con 0x7f24d814be90 2024-03-25T19:27:53.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.563+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754069658 0 0) 0x7f24d80af940 con 0x7f24d80b1730 2024-03-25T19:27:53.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.563+0000 7f24d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:53.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.563+0000 7f24d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:53.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.563+0000 7f24d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:53.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.563+0000 7f24d6ffd640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:27:53.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.563+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d80b0a70 con 0x7f24d80b1730 2024-03-25T19:27:53.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.564+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778847783 0 0) 0x7f24d80b0d50 con 0x7f24d80b4d60 2024-03-25T19:27:53.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.564+0000 7f24d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:53.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.564+0000 7f24d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:53.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.564+0000 7f24d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:53.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.564+0000 7f24d6ffd640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:27:53.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.564+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d80af940 con 0x7f24d80b4d60 2024-03-25T19:27:53.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2025180244 0 0) 0x7f24d80b0a70 con 0x7f24d80b1730 2024-03-25T19:27:53.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 >> v1:172.21.15.52:6790/0 conn(0x7f24d80b4d60 legacy=0x7f24d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 >> v1:172.21.15.179:6789/0 conn(0x7f24d814be90 legacy=0x7f24d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:53.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24d6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:27:53.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:53.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:27:53.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d80b1030 con 0x7f24d80b1730 2024-03-25T19:27:53.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:27:53.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:53.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:53.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24de40f640 5 monclient: authenticate success, global_id 4212 2024-03-25T19:27:53.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.567+0000 7f24de40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:53.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f24cc001630 con 0x7f24d80b1730 2024-03-25T19:27:53.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:53.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:53.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24d6ffd640 10 monclient: dump: 2024-03-25T19:27:53.569 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:53.569 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:53.569 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24cc001880 con 0x7f24d80b1730 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24de40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24d6ffd640 1 -- 172.21.15.52:0/2080767325 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f24cc001c50 con 0x7f24d80b1730 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24d6ffd640 10 monclient: dump: 2024-03-25T19:27:53.570 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24de40f640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.569+0000 7f24de40f640 4 set_mon_vals no callback set 2024-03-25T19:27:53.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.570+0000 7f24de40f640 10 monclient: shutdown 2024-03-25T19:27:53.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.570+0000 7f24de40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:53.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.570+0000 7f24de40f640 1 -- 172.21.15.52:0/2080767325 >> v1:172.21.15.52:6789/0 conn(0x7f24d80b1730 legacy=0x7f24d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.570+0000 7f24de40f640 1 -- 172.21.15.52:0/2080767325 shutdown_connections 2024-03-25T19:27:53.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.570+0000 7f24de40f640 1 -- 172.21.15.52:0/2080767325 >> 172.21.15.52:0/2080767325 conn(0x7f24d80617a0 msgr2=0x7f24d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:53.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.570+0000 7f24de40f640 1 -- 172.21.15.52:0/2080767325 shutdown_connections 2024-03-25T19:27:53.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.570+0000 7f24de40f640 1 -- 172.21.15.52:0/2080767325 wait complete. 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.571+0000 7f24de40f640 10 monclient: build_initial_monmap 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.571+0000 7f24de40f640 10 monclient: monmap: 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.572+0000 7f24de40f640 1 librados: starting msgr at 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.572+0000 7f24de40f640 1 librados: starting objecter 2024-03-25T19:27:53.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.572+0000 7f24de40f640 1 Processor -- start 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.572+0000 7f24de40f640 1 -- start start 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.572+0000 7f24de40f640 1 librados: setting wanted keys 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.572+0000 7f24de40f640 1 librados: calling monclient init 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.572+0000 7f24de40f640 10 monclient: init 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.573+0000 7f24de40f640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.573+0000 7f24de40f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.573+0000 7f24de40f640 10 monclient(hunting): picked mon.noname-a con 0x7f24d80b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.573+0000 7f24de40f640 10 monclient(hunting): picked mon.noname-c con 0x7f24d80b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.573+0000 7f24de40f640 10 monclient(hunting): picked mon.noname-b con 0x7f24d814be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.573+0000 7f24de40f640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d813a920 con 0x7f24d80b1730 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.573+0000 7f24de40f640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d8138520 con 0x7f24d814be90 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.573+0000 7f24de40f640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d81381b0 con 0x7f24d80b4d60 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.573+0000 7f24de40f640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.573+0000 7f24de40f640 10 monclient(hunting): authenticate will time out at 5921.753561s 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.574+0000 7f24d7fff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f24d80b1730 0x7f24d8131d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:46758/0 (socket says 172.21.15.52:46758) 2024-03-25T19:27:53.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.574+0000 7f24d7fff640 1 -- 172.21.15.52:0/2351782518 learned_addr learned my addr 172.21.15.52:0/2351782518 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:27:53.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.574+0000 7f24d4ff9640 10 client.?.objecter ms_handle_connect 0x7f24d80b1730 2024-03-25T19:27:53.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.574+0000 7f24d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:53.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.574+0000 7f24d4ff9640 10 client.?.objecter ms_handle_connect 0x7f24d80b4d60 2024-03-25T19:27:53.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.574+0000 7f24d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:53.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.574+0000 7f24d4ff9640 10 client.?.objecter ms_handle_connect 0x7f24d814be90 2024-03-25T19:27:53.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.574+0000 7f24d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:53.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.576+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594943478 0 0) 0x7f24d81381b0 con 0x7f24d80b4d60 2024-03-25T19:27:53.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.576+0000 7f24d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:53.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.576+0000 7f24d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:53.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.576+0000 7f24d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:53.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.576+0000 7f24d4ff9640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:27:53.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.576+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c4004eb0 con 0x7f24d80b4d60 2024-03-25T19:27:53.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846214952 0 0) 0x7f24d813a920 con 0x7f24d80b1730 2024-03-25T19:27:53.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:53.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:53.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:53.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:27:53.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d81381b0 con 0x7f24d80b1730 2024-03-25T19:27:53.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059614280 0 0) 0x7f24d8138520 con 0x7f24d814be90 2024-03-25T19:27:53.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:53.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:53.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:53.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:27:53.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.577+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d813a920 con 0x7f24d814be90 2024-03-25T19:27:53.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.582+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2658564790 0 0) 0x7f24c4004eb0 con 0x7f24d80b4d60 2024-03-25T19:27:53.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.582+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24d8138520 con 0x7f24d80b4d60 2024-03-25T19:27:53.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.582+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f24cc001490 con 0x7f24d80b4d60 2024-03-25T19:27:53.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.582+0000 7f24d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:53.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.582+0000 7f24d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.582+0000 7f24d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:53.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.582+0000 7f24d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:53.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.584+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1649361931 0 0) 0x7f24d81381b0 con 0x7f24d80b1730 2024-03-25T19:27:53.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.584+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24c4004eb0 con 0x7f24d80b1730 2024-03-25T19:27:53.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.584+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2796443279 0 0) 0x7f24d813a920 con 0x7f24d814be90 2024-03-25T19:27:53.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.584+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24d81381b0 con 0x7f24d814be90 2024-03-25T19:27:53.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.584+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f24d0001a30 con 0x7f24d814be90 2024-03-25T19:27:53.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.584+0000 7f24d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:53.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.584+0000 7f24d4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:53.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.584+0000 7f24d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:53.584 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:53.584 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:53.584 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:53.584 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:53.584 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:53.585 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:53.585 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:53.585 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:53.585 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:53.585 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:53.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3437481254 0 0) 0x7f24d8138520 con 0x7f24d80b4d60 2024-03-25T19:27:53.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 >> v1:172.21.15.179:6789/0 conn(0x7f24d814be90 legacy=0x7f24d81e2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 >> v1:172.21.15.52:6789/0 conn(0x7f24d80b1730 legacy=0x7f24d8131d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24d4ff9640 1 monclient: found mon.c 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24d4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d8139a00 con 0x7f24d80b4d60 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24d4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24de40f640 5 monclient: authenticate success, global_id 4211 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24de40f640 10 monclient: _renew_subs 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24de40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24de40f640 1 -- 172.21.15.52:0/2351782518 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d8138f60 con 0x7f24d80b4d60 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24de40f640 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:53.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24de40f640 10 monclient: _renew_subs 2024-03-25T19:27:53.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24de40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:27:53.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24de40f640 1 -- 172.21.15.52:0/2351782518 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d8132490 con 0x7f24d80b4d60 2024-03-25T19:27:53.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.586+0000 7f24de40f640 1 librados: init done 2024-03-25T19:27:53.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.587+0000 7f24de40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:53.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.587+0000 7f24de40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:53.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.587+0000 7f24de40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:27:53.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.587+0000 7f24de40f640 1 -- 172.21.15.52:0/2351782518 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d81401f0 con 0x7f24d80b4d60 2024-03-25T19:27:53.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.724+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24cc0016b0 con 0x7f24d80b4d60 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.724+0000 7f24d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.724+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f24cc004d90 con 0x7f24d80b4d60 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.724+0000 7f24d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.724+0000 7f24d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.724+0000 7f24d4ff9640 10 monclient: dump: 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:27:53.724 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:27:53.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.724+0000 7f24d67fc640 4 set_mon_vals no callback set 2024-03-25T19:27:53.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.724+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f24cc01ad60 con 0x7f24d80b4d60 2024-03-25T19:27:53.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.725+0000 7f24d7fff640 10 monclient: get_auth_request con 0x7f24c406e220 auth_method 0 2024-03-25T19:27:53.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.725+0000 7f24d4ff9640 10 client.4211.objecter ms_handle_connect 0x7f24c406e220 2024-03-25T19:27:53.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.804+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f24cc054970 con 0x7f24d80b4d60 2024-03-25T19:27:53.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.804+0000 7f24d4ff9640 10 client.4211.objecter ms_dispatch 0x7f24d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:53.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.804+0000 7f24d4ff9640 3 client.4211.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:53.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.804+0000 7f24d4ff9640 3 client.4211.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:53.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:53.804+0000 7f24d4ff9640 20 client.4211.objecter dump_active .. 0 homeless 2024-03-25T19:27:54.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.059+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24cc050500 con 0x7f24d80b4d60 2024-03-25T19:27:54.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.059+0000 7f24d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:54.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.059+0000 7f24d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.327+0000 7f24de40f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.328+0000 7f24de40f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:54.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.328+0000 7f24de40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:27:54.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.328+0000 7f24de40f640 1 -- 172.21.15.52:0/2351782518 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24d81386b0 con 0x7f24d80b4d60 2024-03-25T19:27:54.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.430+0000 7f24d4ff9640 1 -- 172.21.15.52:0/2351782518 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f24cc0523b0 con 0x7f24d80b4d60 2024-03-25T19:27:54.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.430+0000 7f24d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:54.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.430+0000 7f24d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:54.430 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:27:54.430 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:54.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.432+0000 7f24de40f640 10 librados: watch_flush enter 2024-03-25T19:27:54.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.432+0000 7f24de40f640 10 librados: watch_flush exit 2024-03-25T19:27:54.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.432+0000 7f24de40f640 20 client.4211.objecter shutdown clearing up homeless session... 2024-03-25T19:27:54.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.432+0000 7f24de40f640 10 client.4211.objecter successfully canceled tick 2024-03-25T19:27:54.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.433+0000 7f24de40f640 1 -- 172.21.15.52:0/2351782518 >> v1:172.21.15.52:6800/3967427147 conn(0x7f24c406e220 legacy=0x7f24c40706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:54.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.433+0000 7f24de40f640 10 monclient: shutdown 2024-03-25T19:27:54.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.433+0000 7f24de40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:54.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.433+0000 7f24de40f640 1 -- 172.21.15.52:0/2351782518 >> v1:172.21.15.52:6790/0 conn(0x7f24d80b4d60 legacy=0x7f24d81d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:54.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.433+0000 7f24de40f640 1 -- 172.21.15.52:0/2351782518 shutdown_connections 2024-03-25T19:27:54.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.433+0000 7f24de40f640 1 -- 172.21.15.52:0/2351782518 >> 172.21.15.52:0/2351782518 conn(0x7f24d80617a0 msgr2=0x7f24d80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:54.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.433+0000 7f24de40f640 1 -- 172.21.15.52:0/2351782518 shutdown_connections 2024-03-25T19:27:54.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.433+0000 7f24de40f640 1 -- 172.21.15.52:0/2351782518 wait complete. 2024-03-25T19:27:54.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:27:54.433+0000 7f24de40f640 1 librados: shutdown 2024-03-25T19:27:54.447 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:54.463 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:00:25.135 35760== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:55.272 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:00:25.948 35758== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:55.313 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:00:25.962 35761== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:55.451 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:00:26.123 35759== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:56.051 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:00:26.729 35418== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:56.053 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:00:26.717 35419== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:56.330 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:00:26.990 35420== Warning: unimplemented fcntl command: 1038 2024-03-25T19:27:56.386 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:00:27.040 35421== Warning: unimplemented fcntl command: 1038 2024-03-25T19:28:01.254 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:01.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.348+0000 7f7020640640 10 monclient: get_monmap_and_config 2024-03-25T19:28:01.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.348+0000 7f7020640640 10 monclient: build_initial_monmap 2024-03-25T19:28:01.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.348+0000 7f7020640640 10 monclient: monmap: 2024-03-25T19:28:01.348 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:01.348 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:01.348 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:28:01.348803+0000 2024-03-25T19:28:01.348 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:28:01.348803+0000 2024-03-25T19:28:01.348 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:01.349 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:01.349 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:01.349 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:28:01.349 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:28:01.349 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:01.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.348+0000 7f7020640640 1 Processor -- start 2024-03-25T19:28:01.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.348+0000 7f7020640640 1 -- start start 2024-03-25T19:28:01.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.348+0000 7f7020640640 10 monclient: init 2024-03-25T19:28:01.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.349+0000 7f7020640640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:01.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.349+0000 7f7020640640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:01.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.349+0000 7f7020640640 10 monclient(hunting): picked mon.noname-c con 0x7f70180b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:01.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.349+0000 7f7020640640 10 monclient(hunting): picked mon.noname-b con 0x7f70180b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:01.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.349+0000 7f7020640640 10 monclient(hunting): picked mon.noname-a con 0x7f701814be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:01.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.349+0000 7f7020640640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70180af940 con 0x7f701814be90 2024-03-25T19:28:01.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.349+0000 7f7020640640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70180b0a70 con 0x7f70180b4d60 2024-03-25T19:28:01.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.349+0000 7f7020640640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70180b0d50 con 0x7f70180b1730 2024-03-25T19:28:01.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.349+0000 7f7020640640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:01.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.349+0000 7f7020640640 10 monclient(hunting): authenticate will time out at 5929.529712s 2024-03-25T19:28:01.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.350+0000 7f701ebdd640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f701814be90 0x7f701814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:51192/0 (socket says 172.21.15.52:51192) 2024-03-25T19:28:01.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.350+0000 7f701ebdd640 1 -- 172.21.15.52:0/2823551499 learned_addr learned my addr 172.21.15.52:0/2823551499 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:01.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675068453 0 0) 0x7f70180af940 con 0x7f701814be90 2024-03-25T19:28:01.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:28:01.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7004003520 con 0x7f701814be90 2024-03-25T19:28:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881173750 0 0) 0x7f70180b0d50 con 0x7f70180b1730 2024-03-25T19:28:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:28:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.353+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70180af940 con 0x7f70180b1730 2024-03-25T19:28:01.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.354+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540439696 0 0) 0x7f70180b0a70 con 0x7f70180b4d60 2024-03-25T19:28:01.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.354+0000 7f701d3da640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.354+0000 7f701d3da640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.354+0000 7f701d3da640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.354+0000 7f701d3da640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:28:01.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.354+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70180b0d50 con 0x7f70180b4d60 2024-03-25T19:28:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3815544233 0 0) 0x7f7004003520 con 0x7f701814be90 2024-03-25T19:28:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 >> v1:172.21.15.52:6790/0 conn(0x7f70180b1730 legacy=0x7f70180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 >> v1:172.21.15.179:6789/0 conn(0x7f70180b4d60 legacy=0x7f701813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 10 monclient: _finish_hunting 0 2024-03-25T19:28:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 1 monclient: found mon.noname-a 2024-03-25T19:28:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:28:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70180b1030 con 0x7f701814be90 2024-03-25T19:28:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 10 monclient: _finish_auth 0 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 10 monclient: _check_auth_tickets 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f7020640640 5 monclient: authenticate success, global_id 4218 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f7020640640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f70100019d0 con 0x7f701814be90 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f701d3da640 10 monclient: dump: 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:01.359 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:01.360 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:01.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.358+0000 7f7020640640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:01.361 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.360+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7010001c20 con 0x7f701814be90 2024-03-25T19:28:01.361 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.360+0000 7f701d3da640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:01.361 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.360+0000 7f701d3da640 1 -- 172.21.15.52:0/2823551499 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7010001fb0 con 0x7f701814be90 2024-03-25T19:28:01.361 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.360+0000 7f701d3da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:01.361 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.360+0000 7f701d3da640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.360+0000 7f701d3da640 10 monclient: dump: 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.360+0000 7f7020640640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.360+0000 7f7020640640 4 set_mon_vals no callback set 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.361+0000 7f7020640640 10 monclient: shutdown 2024-03-25T19:28:01.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.361+0000 7f7020640640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:01.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.361+0000 7f7020640640 1 -- 172.21.15.52:0/2823551499 >> v1:172.21.15.52:6789/0 conn(0x7f701814be90 legacy=0x7f701814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.361+0000 7f7020640640 1 -- 172.21.15.52:0/2823551499 shutdown_connections 2024-03-25T19:28:01.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.361+0000 7f7020640640 1 -- 172.21.15.52:0/2823551499 >> 172.21.15.52:0/2823551499 conn(0x7f70180617a0 msgr2=0x7f7018061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:01.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.362+0000 7f7020640640 1 -- 172.21.15.52:0/2823551499 shutdown_connections 2024-03-25T19:28:01.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.362+0000 7f7020640640 1 -- 172.21.15.52:0/2823551499 wait complete. 2024-03-25T19:28:01.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.362+0000 7f7020640640 10 monclient: build_initial_monmap 2024-03-25T19:28:01.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.362+0000 7f7020640640 10 monclient: monmap: 2024-03-25T19:28:01.365 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:01.365 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:01.365 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:28:01.365 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:28:01.365 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.363+0000 7f7020640640 1 librados: starting msgr at 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.363+0000 7f7020640640 1 librados: starting objecter 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.363+0000 7f7020640640 1 Processor -- start 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.363+0000 7f7020640640 1 -- start start 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.363+0000 7f7020640640 1 librados: setting wanted keys 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.363+0000 7f7020640640 1 librados: calling monclient init 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.363+0000 7f7020640640 10 monclient: init 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f7020640640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f7020640640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f7020640640 10 monclient(hunting): picked mon.noname-c con 0x7f70180b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f7020640640 10 monclient(hunting): picked mon.noname-b con 0x7f70180b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f7020640640 10 monclient(hunting): picked mon.noname-a con 0x7f701814be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f7020640640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70181d7c80 con 0x7f701814be90 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f7020640640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7018143200 con 0x7f70180b4d60 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f7020640640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70181d7830 con 0x7f70180b1730 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f7020640640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f7020640640 10 monclient(hunting): authenticate will time out at 5929.544476s 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f701ebdd640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f701814be90 0x7f7018141650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:51202/0 (socket says 172.21.15.52:51202) 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.364+0000 7f701ebdd640 1 -- 172.21.15.52:0/3918095717 learned_addr learned my addr 172.21.15.52:0/3918095717 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.365+0000 7f700effd640 10 client.?.objecter ms_handle_connect 0x7f701814be90 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.365+0000 7f700effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.365+0000 7f700effd640 10 client.?.objecter ms_handle_connect 0x7f70180b4d60 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.365+0000 7f700effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.365+0000 7f700effd640 10 client.?.objecter ms_handle_connect 0x7f70180b1730 2024-03-25T19:28:01.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.365+0000 7f700effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254703811 0 0) 0x7f70181d7c80 con 0x7f701814be90 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ffc004eb0 con 0x7f701814be90 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790883227 0 0) 0x7f70181d7830 con 0x7f70180b1730 2024-03-25T19:28:01.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:28:01.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70181d7c80 con 0x7f70180b1730 2024-03-25T19:28:01.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872056086 0 0) 0x7f7018143200 con 0x7f70180b4d60 2024-03-25T19:28:01.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:01.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:01.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:01.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:28:01.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.368+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70181d7830 con 0x7f70180b4d60 2024-03-25T19:28:01.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.375+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 374233068 0 0) 0x7f70181d7c80 con 0x7f70180b1730 2024-03-25T19:28:01.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.375+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7018143200 con 0x7f70180b1730 2024-03-25T19:28:01.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.375+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7014001870 con 0x7f70180b1730 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.375+0000 7f700effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.375+0000 7f700effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.375+0000 7f700effd640 10 monclient(hunting): dump: 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.375+0000 7f700effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.375+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1600047089 0 0) 0x7f6ffc004eb0 con 0x7f701814be90 2024-03-25T19:28:01.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.375+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70181d7c80 con 0x7f701814be90 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.375+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1878477073 0 0) 0x7f70181d7830 con 0x7f70180b4d60 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.376+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ffc004eb0 con 0x7f70180b4d60 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.376+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f70080018b0 con 0x7f70180b4d60 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.376+0000 7f700effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.376+0000 7f700effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.376+0000 7f700effd640 10 monclient(hunting): dump: 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:01.376 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:01.377 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:01.377 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:01.377 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:01.377 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:01.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 709398143 0 0) 0x7f7018143200 con 0x7f70180b1730 2024-03-25T19:28:01.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 >> v1:172.21.15.179:6789/0 conn(0x7f70180b4d60 legacy=0x7f70181377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 >> v1:172.21.15.52:6789/0 conn(0x7f701814be90 legacy=0x7f7018141650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:01.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f700effd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:01.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f700effd640 1 monclient: found mon.c 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f700effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f700effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70181d79c0 con 0x7f70180b1730 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f700effd640 10 monclient: _finish_auth 0 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f700effd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f700effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f7020640640 5 monclient: authenticate success, global_id 4217 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f7020640640 10 monclient: _renew_subs 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f7020640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.379+0000 7f7020640640 1 -- 172.21.15.52:0/3918095717 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7018141d60 con 0x7f70180b1730 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.380+0000 7f7020640640 10 client.4217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.380+0000 7f7020640640 10 monclient: _renew_subs 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.380+0000 7f7020640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.380+0000 7f7020640640 1 -- 172.21.15.52:0/3918095717 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7018142010 con 0x7f70180b1730 2024-03-25T19:28:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.380+0000 7f7020640640 1 librados: init done 2024-03-25T19:28:01.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.381+0000 7f7020640640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:01.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.381+0000 7f7020640640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:01.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.381+0000 7f7020640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:28:01.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.381+0000 7f7020640640 1 -- 172.21.15.52:0/3918095717 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7018142ad0 con 0x7f70180b1730 2024-03-25T19:28:01.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.440+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7014003610 con 0x7f70180b1730 2024-03-25T19:28:01.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.440+0000 7f700effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:01.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.440+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7014003920 con 0x7f70180b1730 2024-03-25T19:28:01.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.440+0000 7f700effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:01.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.440+0000 7f700effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:01.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.440+0000 7f700effd640 10 monclient: dump: 2024-03-25T19:28:01.439 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:01.440 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:01.440 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:01.440 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:01.440 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:01.440 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:01.440 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:01.440 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:01.440 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:01.440 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:01.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.440+0000 7f701cbd9640 4 set_mon_vals no callback set 2024-03-25T19:28:01.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.440+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f70140198f0 con 0x7f70180b1730 2024-03-25T19:28:01.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.441+0000 7f701dbdb640 10 monclient: get_auth_request con 0x7f6ffc06e1d0 auth_method 0 2024-03-25T19:28:01.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.441+0000 7f700effd640 10 client.4217.objecter ms_handle_connect 0x7f6ffc06e1d0 2024-03-25T19:28:01.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.833+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f7014053730 con 0x7f70180b1730 2024-03-25T19:28:01.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.833+0000 7f700effd640 10 client.4217.objecter ms_dispatch 0x7f7018000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:01.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.833+0000 7f700effd640 3 client.4217.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:01.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.833+0000 7f700effd640 3 client.4217.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:01.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.833+0000 7f700effd640 20 client.4217.objecter dump_active .. 0 homeless 2024-03-25T19:28:01.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.833+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f701404f240 con 0x7f70180b1730 2024-03-25T19:28:01.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.833+0000 7f700effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:01.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:01.833+0000 7f700effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:02.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.098+0000 7f7020640640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:02.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.098+0000 7f7020640640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:02.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.098+0000 7f7020640640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:28:02.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.098+0000 7f7020640640 1 -- 172.21.15.52:0/3918095717 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70180627b0 con 0x7f70180b1730 2024-03-25T19:28:02.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.114+0000 7f700effd640 1 -- 172.21.15.52:0/3918095717 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f7014051170 con 0x7f70180b1730 2024-03-25T19:28:02.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.114+0000 7f700effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:02.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.114+0000 7f700effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:02.114 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:28:02.114 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:02.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.116+0000 7f7020640640 10 librados: watch_flush enter 2024-03-25T19:28:02.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.116+0000 7f7020640640 10 librados: watch_flush exit 2024-03-25T19:28:02.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.116+0000 7f7020640640 20 client.4217.objecter shutdown clearing up homeless session... 2024-03-25T19:28:02.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.116+0000 7f7020640640 10 client.4217.objecter successfully canceled tick 2024-03-25T19:28:02.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.117+0000 7f7020640640 1 -- 172.21.15.52:0/3918095717 >> v1:172.21.15.52:6800/3967427147 conn(0x7f6ffc06e1d0 legacy=0x7f6ffc070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:02.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.117+0000 7f7020640640 10 monclient: shutdown 2024-03-25T19:28:02.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.117+0000 7f7020640640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:02.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.117+0000 7f7020640640 1 -- 172.21.15.52:0/3918095717 >> v1:172.21.15.52:6790/0 conn(0x7f70180b1730 legacy=0x7f70181d81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:02.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.117+0000 7f7020640640 1 -- 172.21.15.52:0/3918095717 shutdown_connections 2024-03-25T19:28:02.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.117+0000 7f7020640640 1 -- 172.21.15.52:0/3918095717 >> 172.21.15.52:0/3918095717 conn(0x7f70180617a0 msgr2=0x7f70180b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:02.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.117+0000 7f7020640640 1 -- 172.21.15.52:0/3918095717 shutdown_connections 2024-03-25T19:28:02.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.117+0000 7f7020640640 1 -- 172.21.15.52:0/3918095717 wait complete. 2024-03-25T19:28:02.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:02.117+0000 7f7020640640 1 librados: shutdown 2024-03-25T19:28:02.127 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:03.345 INFO:tasks.ceph.osd.5.smithi179.stderr:2024-03-25T19:28:03.341+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:28:03.788 INFO:tasks.ceph.osd.7.smithi179.stderr:2024-03-25T19:28:03.731+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:28:03.855 INFO:tasks.ceph.osd.4.smithi179.stderr:2024-03-25T19:28:03.852+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:28:04.047 INFO:tasks.ceph.osd.6.smithi179.stderr:2024-03-25T19:28:04.044+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:28:04.705 INFO:tasks.ceph.osd.1.smithi052.stderr:2024-03-25T19:28:04.702+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:28:04.761 INFO:tasks.ceph.osd.0.smithi052.stderr:2024-03-25T19:28:04.758+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:28:05.442 INFO:tasks.ceph.osd.3.smithi052.stderr:2024-03-25T19:28:05.441+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:28:05.451 INFO:tasks.ceph.osd.2.smithi052.stderr:2024-03-25T19:28:05.448+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:28:08.933 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.048+0000 7f6d6064a640 10 monclient: get_monmap_and_config 2024-03-25T19:28:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.048+0000 7f6d6064a640 10 monclient: build_initial_monmap 2024-03-25T19:28:09.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.048+0000 7f6d6064a640 10 monclient: monmap: 2024-03-25T19:28:09.049 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:09.049 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:09.049 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:28:09.049775+0000 2024-03-25T19:28:09.049 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:28:09.049775+0000 2024-03-25T19:28:09.049 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:09.049 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:09.049 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:09.049 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:28:09.049 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:28:09.049 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.048+0000 7f6d6064a640 1 Processor -- start 2024-03-25T19:28:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.049+0000 7f6d6064a640 1 -- start start 2024-03-25T19:28:09.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.049+0000 7f6d6064a640 10 monclient: init 2024-03-25T19:28:09.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.050+0000 7f6d6064a640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:09.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.050+0000 7f6d6064a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:09.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.050+0000 7f6d6064a640 10 monclient(hunting): picked mon.noname-a con 0x7f6d580b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:09.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.050+0000 7f6d6064a640 10 monclient(hunting): picked mon.noname-b con 0x7f6d580b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:09.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.050+0000 7f6d6064a640 10 monclient(hunting): picked mon.noname-c con 0x7f6d5814be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:09.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.050+0000 7f6d6064a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d580af940 con 0x7f6d580b4d60 2024-03-25T19:28:09.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.050+0000 7f6d6064a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d580b0a70 con 0x7f6d5814be90 2024-03-25T19:28:09.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.050+0000 7f6d6064a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d580b0d50 con 0x7f6d580b1730 2024-03-25T19:28:09.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.050+0000 7f6d6064a640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:09.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.050+0000 7f6d6064a640 10 monclient(hunting): authenticate will time out at 5937.231468s 2024-03-25T19:28:09.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.056+0000 7f6d5ebe7640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f6d5814be90 0x7f6d5814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:47718/0 (socket says 172.21.15.52:47718) 2024-03-25T19:28:09.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.056+0000 7f6d5ebe7640 1 -- 172.21.15.52:0/3085383107 learned_addr learned my addr 172.21.15.52:0/3085383107 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:09.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.062+0000 7f6d5d3e4640 1 -- 172.21.15.52:0/3085383107 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728878785 0 0) 0x7f6d580b0a70 con 0x7f6d5814be90 2024-03-25T19:28:09.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.062+0000 7f6d5d3e4640 10 monclient(hunting): _negotiate 2024-03-25T19:28:09.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.062+0000 7f6d5d3e4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:09.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.062+0000 7f6d5d3e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:09.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.062+0000 7f6d5d3e4640 10 monclient(hunting): my global_id is 4252 2024-03-25T19:28:09.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.062+0000 7f6d5d3e4640 1 -- 172.21.15.52:0/3085383107 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d44003520 con 0x7f6d5814be90 2024-03-25T19:28:09.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.072+0000 7f6d5d3e4640 1 -- 172.21.15.52:0/3085383107 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962249799 0 0) 0x7f6d580af940 con 0x7f6d580b4d60 2024-03-25T19:28:09.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.072+0000 7f6d5d3e4640 10 monclient(hunting): _negotiate 2024-03-25T19:28:09.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.072+0000 7f6d5d3e4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:09.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.072+0000 7f6d5d3e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:09.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.072+0000 7f6d5d3e4640 10 monclient(hunting): my global_id is 4248 2024-03-25T19:28:09.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.072+0000 7f6d5d3e4640 1 -- 172.21.15.52:0/3085383107 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d580b0a70 con 0x7f6d580b4d60 2024-03-25T19:28:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.077+0000 7f6d5d3e4640 1 -- 172.21.15.52:0/3085383107 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2587793944 0 0) 0x7f6d44003520 con 0x7f6d5814be90 2024-03-25T19:28:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.077+0000 7f6d5d3e4640 1 -- 172.21.15.52:0/3085383107 >> v1:172.21.15.52:6790/0 conn(0x7f6d580b1730 legacy=0x7f6d580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.077+0000 7f6d5d3e4640 1 -- 172.21.15.52:0/3085383107 >> v1:172.21.15.52:6789/0 conn(0x7f6d580b4d60 legacy=0x7f6d5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 10 monclient: _finish_hunting 0 2024-03-25T19:28:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 1 monclient: found mon.noname-c 2024-03-25T19:28:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:28:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 1 -- 172.21.15.52:0/3085383107 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d580b1030 con 0x7f6d5814be90 2024-03-25T19:28:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 10 monclient: _finish_auth 0 2024-03-25T19:28:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 10 monclient: _check_auth_tickets 2024-03-25T19:28:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 1 -- 172.21.15.52:0/3085383107 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6d50001690 con 0x7f6d5814be90 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d5d3e4640 10 monclient: dump: 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d6064a640 5 monclient: authenticate success, global_id 4252 2024-03-25T19:28:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.078+0000 7f6d6064a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:09.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.097+0000 7f6d5d3e4640 1 -- 172.21.15.52:0/3085383107 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d500017f0 con 0x7f6d5814be90 2024-03-25T19:28:09.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.097+0000 7f6d5d3e4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.097+0000 7f6d5d3e4640 1 -- 172.21.15.52:0/3085383107 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6d50001c00 con 0x7f6d5814be90 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.097+0000 7f6d6064a640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.097+0000 7f6d6064a640 4 set_mon_vals no callback set 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.097+0000 7f6d5d3e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.097+0000 7f6d5d3e4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.097+0000 7f6d5d3e4640 10 monclient: dump: 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:09.098 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:09.099 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:09.099 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:09.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.097+0000 7f6d6064a640 10 monclient: shutdown 2024-03-25T19:28:09.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.097+0000 7f6d6064a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:09.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.097+0000 7f6d6064a640 1 -- 172.21.15.52:0/3085383107 >> v1:172.21.15.179:6789/0 conn(0x7f6d5814be90 legacy=0x7f6d5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.098+0000 7f6d6064a640 1 -- 172.21.15.52:0/3085383107 shutdown_connections 2024-03-25T19:28:09.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.098+0000 7f6d6064a640 1 -- 172.21.15.52:0/3085383107 >> 172.21.15.52:0/3085383107 conn(0x7f6d580617a0 msgr2=0x7f6d58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:09.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.098+0000 7f6d6064a640 1 -- 172.21.15.52:0/3085383107 shutdown_connections 2024-03-25T19:28:09.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.098+0000 7f6d6064a640 1 -- 172.21.15.52:0/3085383107 wait complete. 2024-03-25T19:28:09.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.098+0000 7f6d6064a640 10 monclient: build_initial_monmap 2024-03-25T19:28:09.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.098+0000 7f6d6064a640 10 monclient: monmap: 2024-03-25T19:28:09.101 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.099+0000 7f6d6064a640 1 librados: starting msgr at 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.099+0000 7f6d6064a640 1 librados: starting objecter 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.099+0000 7f6d6064a640 1 Processor -- start 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.099+0000 7f6d6064a640 1 -- start start 2024-03-25T19:28:09.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.099+0000 7f6d6064a640 1 librados: setting wanted keys 2024-03-25T19:28:09.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.099+0000 7f6d6064a640 1 librados: calling monclient init 2024-03-25T19:28:09.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.099+0000 7f6d6064a640 10 monclient: init 2024-03-25T19:28:09.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.100+0000 7f6d6064a640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:09.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.100+0000 7f6d6064a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:09.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.100+0000 7f6d6064a640 10 monclient(hunting): picked mon.noname-a con 0x7f6d580b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:09.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.100+0000 7f6d6064a640 10 monclient(hunting): picked mon.noname-b con 0x7f6d580b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:09.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.100+0000 7f6d6064a640 10 monclient(hunting): picked mon.noname-c con 0x7f6d5814be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:09.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.100+0000 7f6d6064a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d58136020 con 0x7f6d580b1730 2024-03-25T19:28:09.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.100+0000 7f6d6064a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d58136b90 con 0x7f6d580b4d60 2024-03-25T19:28:09.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.100+0000 7f6d6064a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d58136d20 con 0x7f6d5814be90 2024-03-25T19:28:09.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.100+0000 7f6d6064a640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:09.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.100+0000 7f6d6064a640 10 monclient(hunting): authenticate will time out at 5937.281390s 2024-03-25T19:28:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.116+0000 7f6d5ebe7640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f6d5814be90 0x7f6d581d7e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:57552/0 (socket says 172.21.15.52:57552) 2024-03-25T19:28:09.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.116+0000 7f6d5ebe7640 1 -- 172.21.15.52:0/2895214134 learned_addr learned my addr 172.21.15.52:0/2895214134 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:09.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.117+0000 7f6d4effd640 10 client.?.objecter ms_handle_connect 0x7f6d5814be90 2024-03-25T19:28:09.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.117+0000 7f6d4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:09.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.123+0000 7f6d4effd640 10 client.?.objecter ms_handle_connect 0x7f6d580b4d60 2024-03-25T19:28:09.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.123+0000 7f6d4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:09.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.128+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270897251 0 0) 0x7f6d58136d20 con 0x7f6d5814be90 2024-03-25T19:28:09.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.128+0000 7f6d4effd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:09.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.128+0000 7f6d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:09.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.128+0000 7f6d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:09.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.128+0000 7f6d4effd640 10 monclient(hunting): my global_id is 4247 2024-03-25T19:28:09.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.128+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d3c004eb0 con 0x7f6d5814be90 2024-03-25T19:28:09.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2408229607 0 0) 0x7f6d3c004eb0 con 0x7f6d5814be90 2024-03-25T19:28:09.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d58136d20 con 0x7f6d5814be90 2024-03-25T19:28:09.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6d50001c80 con 0x7f6d5814be90 2024-03-25T19:28:09.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 10 monclient(hunting): dump: 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:09.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412702346 0 0) 0x7f6d58136b90 con 0x7f6d580b4d60 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.138+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d3c004eb0 con 0x7f6d580b4d60 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.139+0000 7f6d4effd640 10 client.?.objecter ms_handle_connect 0x7f6d580b1730 2024-03-25T19:28:09.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.139+0000 7f6d4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:09.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.146+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 438479598 0 0) 0x7f6d58136d20 con 0x7f6d5814be90 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.146+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 >> v1:172.21.15.179:6789/0 conn(0x7f6d580b4d60 legacy=0x7f6d581d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.146+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 >> v1:172.21.15.52:6789/0 conn(0x7f6d580b1730 legacy=0x7f6d58142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d4effd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d4effd640 1 monclient: found mon.c 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d4effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d58135b00 con 0x7f6d5814be90 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d4effd640 10 monclient: _finish_auth 0 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d4effd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d6064a640 5 monclient: authenticate success, global_id 4247 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d6064a640 10 monclient: _renew_subs 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d6064a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:28:09.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d6064a640 1 -- 172.21.15.52:0/2895214134 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d58135d70 con 0x7f6d5814be90 2024-03-25T19:28:09.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d6064a640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:09.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d6064a640 10 monclient: _renew_subs 2024-03-25T19:28:09.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d6064a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:28:09.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d6064a640 1 -- 172.21.15.52:0/2895214134 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d58143650 con 0x7f6d5814be90 2024-03-25T19:28:09.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.147+0000 7f6d6064a640 1 librados: init done 2024-03-25T19:28:09.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.148+0000 7f6d6064a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:09.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.148+0000 7f6d6064a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:09.150 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.148+0000 7f6d6064a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:28:09.150 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.148+0000 7f6d6064a640 1 -- 172.21.15.52:0/2895214134 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d580627b0 con 0x7f6d5814be90 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.456+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d50004a90 con 0x7f6d5814be90 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.456+0000 7f6d4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.456+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6d50004da0 con 0x7f6d5814be90 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.456+0000 7f6d4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.456+0000 7f6d4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.456+0000 7f6d4effd640 10 monclient: dump: 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:09.457 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.456+0000 7f6d5cbe3640 4 set_mon_vals no callback set 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.457+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f6d5001ad70 con 0x7f6d5814be90 2024-03-25T19:28:09.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.457+0000 7f6d5e3e6640 10 monclient: get_auth_request con 0x7f6d3c06dd40 auth_method 0 2024-03-25T19:28:09.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.458+0000 7f6d4effd640 10 client.4247.objecter ms_handle_connect 0x7f6d3c06dd40 2024-03-25T19:28:09.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.463+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f6d50054800 con 0x7f6d5814be90 2024-03-25T19:28:09.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.463+0000 7f6d4effd640 10 client.4247.objecter ms_dispatch 0x7f6d58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:09.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.463+0000 7f6d4effd640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:09.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.463+0000 7f6d4effd640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:09.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.463+0000 7f6d4effd640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-25T19:28:09.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.463+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d50050310 con 0x7f6d5814be90 2024-03-25T19:28:09.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.463+0000 7f6d4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:09.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.463+0000 7f6d4effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:09.737 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.737+0000 7f6d6064a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:09.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.737+0000 7f6d6064a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:09.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.737+0000 7f6d6064a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:28:09.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.737+0000 7f6d6064a640 1 -- 172.21.15.52:0/2895214134 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d581361b0 con 0x7f6d5814be90 2024-03-25T19:28:09.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.759+0000 7f6d4effd640 1 -- 172.21.15.52:0/2895214134 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f6d50052240 con 0x7f6d5814be90 2024-03-25T19:28:09.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.759+0000 7f6d4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:09.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.759+0000 7f6d4effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:09.760 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:28:09.760 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:09.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.762+0000 7f6d6064a640 10 librados: watch_flush enter 2024-03-25T19:28:09.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.762+0000 7f6d6064a640 10 librados: watch_flush exit 2024-03-25T19:28:09.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.762+0000 7f6d6064a640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-25T19:28:09.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.762+0000 7f6d6064a640 10 client.4247.objecter successfully canceled tick 2024-03-25T19:28:09.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.762+0000 7f6d6064a640 1 -- 172.21.15.52:0/2895214134 >> v1:172.21.15.52:6800/3967427147 conn(0x7f6d3c06dd40 legacy=0x7f6d3c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.762+0000 7f6d6064a640 10 monclient: shutdown 2024-03-25T19:28:09.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.762+0000 7f6d6064a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:09.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.762+0000 7f6d6064a640 1 -- 172.21.15.52:0/2895214134 >> v1:172.21.15.52:6790/0 conn(0x7f6d5814be90 legacy=0x7f6d581d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:09.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.762+0000 7f6d6064a640 1 -- 172.21.15.52:0/2895214134 shutdown_connections 2024-03-25T19:28:09.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.762+0000 7f6d6064a640 1 -- 172.21.15.52:0/2895214134 >> 172.21.15.52:0/2895214134 conn(0x7f6d580617a0 msgr2=0x7f6d580b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:09.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.763+0000 7f6d6064a640 1 -- 172.21.15.52:0/2895214134 shutdown_connections 2024-03-25T19:28:09.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.763+0000 7f6d6064a640 1 -- 172.21.15.52:0/2895214134 wait complete. 2024-03-25T19:28:09.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:09.763+0000 7f6d6064a640 1 librados: shutdown 2024-03-25T19:28:09.774 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:16.581 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:16.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.700+0000 7fd85da63640 10 monclient: get_monmap_and_config 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.700+0000 7fd85da63640 10 monclient: build_initial_monmap 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.700+0000 7fd85da63640 10 monclient: monmap: 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:28:16.701917+0000 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:28:16.701917+0000 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:28:16.701 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.701+0000 7fd85da63640 1 Processor -- start 2024-03-25T19:28:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.701+0000 7fd85da63640 1 -- start start 2024-03-25T19:28:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.701+0000 7fd85da63640 10 monclient: init 2024-03-25T19:28:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.702+0000 7fd85da63640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.702+0000 7fd85da63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.702+0000 7fd85da63640 10 monclient(hunting): picked mon.noname-c con 0x7fd8580a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.702+0000 7fd85da63640 10 monclient(hunting): picked mon.noname-a con 0x7fd85813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:16.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.702+0000 7fd85da63640 10 monclient(hunting): picked mon.noname-b con 0x7fd8581430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:16.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.702+0000 7fd85da63640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd858142520 con 0x7fd85813f710 2024-03-25T19:28:16.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.702+0000 7fd85da63640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8580b28a0 con 0x7fd8580a98f0 2024-03-25T19:28:16.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.702+0000 7fd85da63640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8580b2a30 con 0x7fd8581430c0 2024-03-25T19:28:16.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.702+0000 7fd85da63640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:16.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.702+0000 7fd85da63640 10 monclient(hunting): authenticate will time out at 5944.883135s 2024-03-25T19:28:16.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.707+0000 7fd8577fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd8580a98f0 0x7fd8580a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:49334/0 (socket says 172.21.15.52:49334) 2024-03-25T19:28:16.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.707+0000 7fd8577fe640 1 -- 172.21.15.52:0/14610792 learned_addr learned my addr 172.21.15.52:0/14610792 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:16.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.714+0000 7fd8567fc640 1 -- 172.21.15.52:0/14610792 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159541128 0 0) 0x7fd8580b28a0 con 0x7fd8580a98f0 2024-03-25T19:28:16.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.714+0000 7fd8567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:16.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.714+0000 7fd8567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:16.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.714+0000 7fd8567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:16.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.714+0000 7fd8567fc640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:28:16.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.714+0000 7fd8567fc640 1 -- 172.21.15.52:0/14610792 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd840003520 con 0x7fd8580a98f0 2024-03-25T19:28:16.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.728+0000 7fd8567fc640 1 -- 172.21.15.52:0/14610792 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731230342 0 0) 0x7fd858142520 con 0x7fd85813f710 2024-03-25T19:28:16.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.728+0000 7fd8567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:16.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.728+0000 7fd8567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:16.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.728+0000 7fd8567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:16.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.728+0000 7fd8567fc640 10 monclient(hunting): my global_id is 4254 2024-03-25T19:28:16.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.728+0000 7fd8567fc640 1 -- 172.21.15.52:0/14610792 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8580b28a0 con 0x7fd85813f710 2024-03-25T19:28:16.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 1 -- 172.21.15.52:0/14610792 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4417988 0 0) 0x7fd840003520 con 0x7fd8580a98f0 2024-03-25T19:28:16.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 1 -- 172.21.15.52:0/14610792 >> v1:172.21.15.52:6790/0 conn(0x7fd8581430c0 legacy=0x7fd85814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 1 -- 172.21.15.52:0/14610792 >> v1:172.21.15.52:6789/0 conn(0x7fd85813f710 legacy=0x7fd8581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:16.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 1 monclient: found mon.noname-c 2024-03-25T19:28:16.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:16.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:28:16.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 1 -- 172.21.15.52:0/14610792 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8580b44c0 con 0x7fd8580a98f0 2024-03-25T19:28:16.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:16.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:16.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:16.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 1 -- 172.21.15.52:0/14610792 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd848001690 con 0x7fd8580a98f0 2024-03-25T19:28:16.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:16.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:16.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd8567fc640 10 monclient: dump: 2024-03-25T19:28:16.739 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd85da63640 5 monclient: authenticate success, global_id 4258 2024-03-25T19:28:16.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.737+0000 7fd85da63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:16.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.753+0000 7fd8567fc640 1 -- 172.21.15.52:0/14610792 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8480017f0 con 0x7fd8580a98f0 2024-03-25T19:28:16.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.753+0000 7fd8567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:16.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.753+0000 7fd8567fc640 1 -- 172.21.15.52:0/14610792 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd848001c00 con 0x7fd8580a98f0 2024-03-25T19:28:16.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.753+0000 7fd85da63640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:16.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.753+0000 7fd85da63640 4 set_mon_vals no callback set 2024-03-25T19:28:16.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.753+0000 7fd8567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:16.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.754+0000 7fd8567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:16.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.754+0000 7fd8567fc640 10 monclient: dump: 2024-03-25T19:28:16.755 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:16.755 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:16.755 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:16.755 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:16.755 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:16.755 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:16.755 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:16.755 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:16.755 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:16.755 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.754+0000 7fd85da63640 10 monclient: shutdown 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.754+0000 7fd85da63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.754+0000 7fd85da63640 1 -- 172.21.15.52:0/14610792 >> v1:172.21.15.179:6789/0 conn(0x7fd8580a98f0 legacy=0x7fd8580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.754+0000 7fd85da63640 1 -- 172.21.15.52:0/14610792 shutdown_connections 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.754+0000 7fd85da63640 1 -- 172.21.15.52:0/14610792 >> 172.21.15.52:0/14610792 conn(0x7fd8580b7ea0 msgr2=0x7fd8580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.754+0000 7fd85da63640 1 -- 172.21.15.52:0/14610792 shutdown_connections 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.755+0000 7fd85da63640 1 -- 172.21.15.52:0/14610792 wait complete. 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.755+0000 7fd85da63640 10 monclient: build_initial_monmap 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.755+0000 7fd85da63640 10 monclient: monmap: 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:28:16.757 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:28:16.758 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:16.758 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:28:16.758 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:16.758 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:28:16.758 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:28:16.758 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:16.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.756+0000 7fd85da63640 1 librados: starting msgr at 2024-03-25T19:28:16.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.756+0000 7fd85da63640 1 librados: starting objecter 2024-03-25T19:28:16.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.756+0000 7fd85da63640 1 Processor -- start 2024-03-25T19:28:16.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.756+0000 7fd85da63640 1 -- start start 2024-03-25T19:28:16.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.756+0000 7fd85da63640 1 librados: setting wanted keys 2024-03-25T19:28:16.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.756+0000 7fd85da63640 1 librados: calling monclient init 2024-03-25T19:28:16.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.756+0000 7fd85da63640 10 monclient: init 2024-03-25T19:28:16.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.757+0000 7fd85da63640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:16.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.757+0000 7fd85da63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:16.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.757+0000 7fd85da63640 10 monclient(hunting): picked mon.noname-c con 0x7fd8580a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:16.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.757+0000 7fd85da63640 10 monclient(hunting): picked mon.noname-a con 0x7fd85813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.757+0000 7fd85da63640 10 monclient(hunting): picked mon.noname-b con 0x7fd8581430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.757+0000 7fd85da63640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8581d7380 con 0x7fd85813f710 2024-03-25T19:28:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.757+0000 7fd85da63640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd858139840 con 0x7fd8581430c0 2024-03-25T19:28:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.757+0000 7fd85da63640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8581d8200 con 0x7fd8580a98f0 2024-03-25T19:28:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.757+0000 7fd85da63640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.757+0000 7fd85da63640 10 monclient(hunting): authenticate will time out at 5944.938191s 2024-03-25T19:28:16.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.768+0000 7fd8577fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fd8580a98f0 0x7fd8581d7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:39792/0 (socket says 172.21.15.52:39792) 2024-03-25T19:28:16.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.768+0000 7fd8577fe640 1 -- 172.21.15.52:0/2950246171 learned_addr learned my addr 172.21.15.52:0/2950246171 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:16.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.768+0000 7fd83ffff640 10 client.?.objecter ms_handle_connect 0x7fd8580a98f0 2024-03-25T19:28:16.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.768+0000 7fd83ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:16.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.772+0000 7fd83ffff640 10 client.?.objecter ms_handle_connect 0x7fd8581430c0 2024-03-25T19:28:16.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.772+0000 7fd83ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:16.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.776+0000 7fd83ffff640 10 client.?.objecter ms_handle_connect 0x7fd85813f710 2024-03-25T19:28:16.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.776+0000 7fd83ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:16.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.785+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563698224 0 0) 0x7fd858139840 con 0x7fd8581430c0 2024-03-25T19:28:16.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.785+0000 7fd83ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:16.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.785+0000 7fd83ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:16.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.785+0000 7fd83ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:16.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.785+0000 7fd83ffff640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:28:16.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.785+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd844004eb0 con 0x7fd8581430c0 2024-03-25T19:28:16.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.789+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297375652 0 0) 0x7fd8581d8200 con 0x7fd8580a98f0 2024-03-25T19:28:16.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.789+0000 7fd83ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:16.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.789+0000 7fd83ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:16.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.789+0000 7fd83ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:16.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.789+0000 7fd83ffff640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:28:16.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.789+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd858139840 con 0x7fd8580a98f0 2024-03-25T19:28:16.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.793+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139091219 0 0) 0x7fd8581d7380 con 0x7fd85813f710 2024-03-25T19:28:16.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.793+0000 7fd83ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:16.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.793+0000 7fd83ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:16.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.793+0000 7fd83ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:16.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.793+0000 7fd83ffff640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:28:16.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.793+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8581d8200 con 0x7fd85813f710 2024-03-25T19:28:16.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.809+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1281523542 0 0) 0x7fd844004eb0 con 0x7fd8581430c0 2024-03-25T19:28:16.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.809+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8581d7380 con 0x7fd8581430c0 2024-03-25T19:28:16.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.809+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd850001b90 con 0x7fd8581430c0 2024-03-25T19:28:16.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.809+0000 7fd83ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:16.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.809+0000 7fd83ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:16.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.809+0000 7fd83ffff640 10 monclient(hunting): dump: 2024-03-25T19:28:16.810 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:16.810 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:16.810 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:16.811 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:16.811 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:16.811 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:16.811 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:16.811 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:16.811 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:16.811 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:16.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.809+0000 7fd83ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.810+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 897906219 0 0) 0x7fd8581d8200 con 0x7fd85813f710 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.810+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd844004eb0 con 0x7fd85813f710 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.810+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd84c001930 con 0x7fd85813f710 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.810+0000 7fd83ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.810+0000 7fd83ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.810+0000 7fd83ffff640 10 monclient(hunting): dump: 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:16.812 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:16.813 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:16.813 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:16.813 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:16.813 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:16.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.815+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1566707265 0 0) 0x7fd858139840 con 0x7fd8580a98f0 2024-03-25T19:28:16.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.815+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8581d8200 con 0x7fd8580a98f0 2024-03-25T19:28:16.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.815+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd848001b40 con 0x7fd8580a98f0 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.815+0000 7fd83ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.815+0000 7fd83ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.815+0000 7fd83ffff640 10 monclient(hunting): dump: 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:16.816 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:16.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 996715615 0 0) 0x7fd844004eb0 con 0x7fd85813f710 2024-03-25T19:28:16.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 >> v1:172.21.15.52:6790/0 conn(0x7fd8580a98f0 legacy=0x7fd8581d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 >> v1:172.21.15.179:6789/0 conn(0x7fd8581430c0 legacy=0x7fd8581daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:16.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd83ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:16.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd83ffff640 1 monclient: found mon.a 2024-03-25T19:28:16.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd83ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:16.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd83ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:16.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8581d75d0 con 0x7fd85813f710 2024-03-25T19:28:16.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd83ffff640 10 monclient: _finish_auth 0 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd83ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd83ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd85da63640 5 monclient: authenticate success, global_id 4257 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd85da63640 10 monclient: _renew_subs 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd85da63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd85da63640 1 -- 172.21.15.52:0/2950246171 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8581db6a0 con 0x7fd85813f710 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd85da63640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd85da63640 10 monclient: _renew_subs 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd85da63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd85da63640 1 -- 172.21.15.52:0/2950246171 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd858137910 con 0x7fd85813f710 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.825+0000 7fd85da63640 1 librados: init done 2024-03-25T19:28:16.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.826+0000 7fd85da63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:16.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.826+0000 7fd85da63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:16.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.826+0000 7fd85da63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:16.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:16.826+0000 7fd85da63640 1 -- 172.21.15.52:0/2950246171 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8580627b0 con 0x7fd85813f710 2024-03-25T19:28:17.150 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.150+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd84c003610 con 0x7fd85813f710 2024-03-25T19:28:17.150 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.150+0000 7fd83ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:17.150 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.150+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd84c001d80 con 0x7fd85813f710 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.150+0000 7fd83ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.150+0000 7fd83ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.150+0000 7fd83ffff640 10 monclient: dump: 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:17.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.150+0000 7fd855ffb640 4 set_mon_vals no callback set 2024-03-25T19:28:17.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.150+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fd84c0197d0 con 0x7fd85813f710 2024-03-25T19:28:17.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.151+0000 7fd8577fe640 10 monclient: get_auth_request con 0x7fd84406e160 auth_method 0 2024-03-25T19:28:17.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.151+0000 7fd83ffff640 10 client.4257.objecter ms_handle_connect 0x7fd84406e160 2024-03-25T19:28:17.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.158+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fd84c053530 con 0x7fd85813f710 2024-03-25T19:28:17.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.159+0000 7fd83ffff640 10 client.4257.objecter ms_dispatch 0x7fd858000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:17.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.159+0000 7fd83ffff640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:17.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.159+0000 7fd83ffff640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:17.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.159+0000 7fd83ffff640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-25T19:28:17.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.159+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd84c04ee70 con 0x7fd85813f710 2024-03-25T19:28:17.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.159+0000 7fd83ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:17.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.159+0000 7fd83ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:17.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.391+0000 7fd85da63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:17.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.391+0000 7fd85da63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:17.392 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.391+0000 7fd85da63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:17.392 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.391+0000 7fd85da63640 1 -- 172.21.15.52:0/2950246171 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd858069f40 con 0x7fd85813f710 2024-03-25T19:28:17.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.416+0000 7fd83ffff640 1 -- 172.21.15.52:0/2950246171 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fd84c050f70 con 0x7fd85813f710 2024-03-25T19:28:17.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.416+0000 7fd83ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:17.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.416+0000 7fd83ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:17.416 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:28:17.417 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:17.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 10 librados: watch_flush enter 2024-03-25T19:28:17.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 10 librados: watch_flush exit 2024-03-25T19:28:17.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-25T19:28:17.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 10 client.4257.objecter successfully canceled tick 2024-03-25T19:28:17.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 1 -- 172.21.15.52:0/2950246171 >> v1:172.21.15.52:6800/3967427147 conn(0x7fd84406e160 legacy=0x7fd844070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:17.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 10 monclient: shutdown 2024-03-25T19:28:17.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:17.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 1 -- 172.21.15.52:0/2950246171 >> v1:172.21.15.52:6789/0 conn(0x7fd85813f710 legacy=0x7fd8581da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:17.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 1 -- 172.21.15.52:0/2950246171 shutdown_connections 2024-03-25T19:28:17.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 1 -- 172.21.15.52:0/2950246171 >> 172.21.15.52:0/2950246171 conn(0x7fd8580b7ea0 msgr2=0x7fd858074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:17.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 1 -- 172.21.15.52:0/2950246171 shutdown_connections 2024-03-25T19:28:17.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 1 -- 172.21.15.52:0/2950246171 wait complete. 2024-03-25T19:28:17.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:17.418+0000 7fd85da63640 1 librados: shutdown 2024-03-25T19:28:17.431 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:20.900 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:01:14.499 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:20.959 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:01:14.558 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:21.013 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:01:14.555 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:21.080 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:01:14.622 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:21.121 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:01:14.720 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:21.205 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:01:14.804 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:21.235 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:01:14.834 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:21.302 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:01:14.844 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:21.456 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:01:15.055 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:24.237 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:24.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.353+0000 7f8aa1d2b640 10 monclient: get_monmap_and_config 2024-03-25T19:28:24.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.353+0000 7f8aa1d2b640 10 monclient: build_initial_monmap 2024-03-25T19:28:24.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.353+0000 7f8aa1d2b640 10 monclient: monmap: 2024-03-25T19:28:24.354 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:24.354 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:24.355 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:28:24.355366+0000 2024-03-25T19:28:24.355 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:28:24.355366+0000 2024-03-25T19:28:24.355 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:24.355 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:24.355 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:24.355 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:28:24.355 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:28:24.355 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:24.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.354+0000 7f8aa1d2b640 1 Processor -- start 2024-03-25T19:28:24.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.354+0000 7f8aa1d2b640 1 -- start start 2024-03-25T19:28:24.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.354+0000 7f8aa1d2b640 10 monclient: init 2024-03-25T19:28:24.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.355+0000 7f8aa1d2b640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:24.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.355+0000 7f8aa1d2b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:24.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.355+0000 7f8aa1d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f8a9c0a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:24.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.355+0000 7f8aa1d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f8a9c13f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:24.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.355+0000 7f8aa1d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f8a9c1430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:24.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.355+0000 7f8aa1d2b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c142520 con 0x7f8a9c0a98f0 2024-03-25T19:28:24.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.355+0000 7f8aa1d2b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c0b28a0 con 0x7f8a9c13f710 2024-03-25T19:28:24.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.355+0000 7f8aa1d2b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c0b2a30 con 0x7f8a9c1430c0 2024-03-25T19:28:24.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.355+0000 7f8aa1d2b640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:24.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.355+0000 7f8aa1d2b640 10 monclient(hunting): authenticate will time out at 5952.536383s 2024-03-25T19:28:24.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.358+0000 7f8a9b7fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f8a9c0a98f0 0x7f8a9c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:55006/0 (socket says 172.21.15.52:55006) 2024-03-25T19:28:24.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.358+0000 7f8a9b7fe640 1 -- 172.21.15.52:0/4084516611 learned_addr learned my addr 172.21.15.52:0/4084516611 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:24.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.377+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380225939 0 0) 0x7f8a9c142520 con 0x7f8a9c0a98f0 2024-03-25T19:28:24.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.377+0000 7f8a9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.377+0000 7f8a9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.377+0000 7f8a9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.377+0000 7f8a9a7fc640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:28:24.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.377+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a84003520 con 0x7f8a9c0a98f0 2024-03-25T19:28:24.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.389+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854280485 0 0) 0x7f8a9c0b28a0 con 0x7f8a9c13f710 2024-03-25T19:28:24.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.389+0000 7f8a9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.389+0000 7f8a9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.389+0000 7f8a9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.389+0000 7f8a9a7fc640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:28:24.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.389+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a9c142520 con 0x7f8a9c13f710 2024-03-25T19:28:24.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.390+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435240596 0 0) 0x7f8a9c0b2a30 con 0x7f8a9c1430c0 2024-03-25T19:28:24.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.390+0000 7f8a9a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.390+0000 7f8a9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.390+0000 7f8a9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.390+0000 7f8a9a7fc640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:28:24.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.390+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a9c0b28a0 con 0x7f8a9c1430c0 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 376408030 0 0) 0x7f8a84003520 con 0x7f8a9c0a98f0 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 >> v1:172.21.15.52:6790/0 conn(0x7f8a9c1430c0 legacy=0x7f8a9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 >> v1:172.21.15.179:6789/0 conn(0x7f8a9c13f710 legacy=0x7f8a9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c0b44c0 con 0x7f8a9c0a98f0 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8a8c001690 con 0x7f8a9c0a98f0 2024-03-25T19:28:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8a9a7fc640 10 monclient: dump: 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8aa1d2b640 5 monclient: authenticate success, global_id 4260 2024-03-25T19:28:24.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.403+0000 7f8aa1d2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:24.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.423+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a8c0017f0 con 0x7f8a9c0a98f0 2024-03-25T19:28:24.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.423+0000 7f8a9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.423+0000 7f8a9a7fc640 1 -- 172.21.15.52:0/4084516611 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8a8c001c00 con 0x7f8a9c0a98f0 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.423+0000 7f8aa1d2b640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.423+0000 7f8aa1d2b640 4 set_mon_vals no callback set 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.423+0000 7f8a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.423+0000 7f8a9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.423+0000 7f8a9a7fc640 10 monclient: dump: 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.424 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:24.425 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:24.425 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:24.425 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:24.425 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:24.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.424+0000 7f8aa1d2b640 10 monclient: shutdown 2024-03-25T19:28:24.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.424+0000 7f8aa1d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:24.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.424+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/4084516611 >> v1:172.21.15.52:6789/0 conn(0x7f8a9c0a98f0 legacy=0x7f8a9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.424+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/4084516611 shutdown_connections 2024-03-25T19:28:24.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.424+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/4084516611 >> 172.21.15.52:0/4084516611 conn(0x7f8a9c0b7ea0 msgr2=0x7f8a9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.424+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/4084516611 shutdown_connections 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.424+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/4084516611 wait complete. 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.425+0000 7f8aa1d2b640 10 monclient: build_initial_monmap 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.425+0000 7f8aa1d2b640 10 monclient: monmap: 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:24.428 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.425+0000 7f8aa1d2b640 1 librados: starting msgr at 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.425+0000 7f8aa1d2b640 1 librados: starting objecter 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.425+0000 7f8aa1d2b640 1 Processor -- start 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.425+0000 7f8aa1d2b640 1 -- start start 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.425+0000 7f8aa1d2b640 1 librados: setting wanted keys 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.425+0000 7f8aa1d2b640 1 librados: calling monclient init 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.425+0000 7f8aa1d2b640 10 monclient: init 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.426+0000 7f8aa1d2b640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.426+0000 7f8aa1d2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.426+0000 7f8aa1d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f8a9c0a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.426+0000 7f8aa1d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f8a9c13f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.426+0000 7f8aa1d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f8a9c1430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.426+0000 7f8aa1d2b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c1db2f0 con 0x7f8a9c13f710 2024-03-25T19:28:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.426+0000 7f8aa1d2b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c139180 con 0x7f8a9c0a98f0 2024-03-25T19:28:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.426+0000 7f8aa1d2b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c137670 con 0x7f8a9c1430c0 2024-03-25T19:28:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.426+0000 7f8aa1d2b640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.426+0000 7f8aa1d2b640 10 monclient(hunting): authenticate will time out at 5952.607499s 2024-03-25T19:28:24.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.432+0000 7f8a9affd640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f8a9c13f710 0x7f8a9c132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:55052/0 (socket says 172.21.15.52:55052) 2024-03-25T19:28:24.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.432+0000 7f8a9affd640 1 -- 172.21.15.52:0/1237772950 learned_addr learned my addr 172.21.15.52:0/1237772950 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:24.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.436+0000 7f8a83fff640 10 client.?.objecter ms_handle_connect 0x7f8a9c13f710 2024-03-25T19:28:24.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.436+0000 7f8a83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:24.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.441+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546766384 0 0) 0x7f8a9c1db2f0 con 0x7f8a9c13f710 2024-03-25T19:28:24.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.441+0000 7f8a83fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.441+0000 7f8a83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.441+0000 7f8a83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.441+0000 7f8a83fff640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:28:24.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.441+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a88004eb0 con 0x7f8a9c13f710 2024-03-25T19:28:24.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.444+0000 7f8a83fff640 10 client.?.objecter ms_handle_connect 0x7f8a9c1430c0 2024-03-25T19:28:24.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.444+0000 7f8a83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:24.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.444+0000 7f8a83fff640 10 client.?.objecter ms_handle_connect 0x7f8a9c0a98f0 2024-03-25T19:28:24.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.444+0000 7f8a83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:24.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.456+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228002480 0 0) 0x7f8a9c137670 con 0x7f8a9c1430c0 2024-03-25T19:28:24.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.456+0000 7f8a83fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.456+0000 7f8a83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.456+0000 7f8a83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.456+0000 7f8a83fff640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:28:24.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.456+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a9c1db2f0 con 0x7f8a9c1430c0 2024-03-25T19:28:24.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.461+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578043438 0 0) 0x7f8a9c139180 con 0x7f8a9c0a98f0 2024-03-25T19:28:24.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.461+0000 7f8a83fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:24.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.461+0000 7f8a83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:24.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.461+0000 7f8a83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:24.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.461+0000 7f8a83fff640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:28:24.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.461+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a9c137670 con 0x7f8a9c0a98f0 2024-03-25T19:28:24.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.463+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 894306581 0 0) 0x7f8a88004eb0 con 0x7f8a9c13f710 2024-03-25T19:28:24.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.463+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a9c139180 con 0x7f8a9c13f710 2024-03-25T19:28:24.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.463+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8a90001a30 con 0x7f8a9c13f710 2024-03-25T19:28:24.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.463+0000 7f8a83fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.463+0000 7f8a83fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.463+0000 7f8a83fff640 10 monclient(hunting): dump: 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:24.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.463+0000 7f8a83fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2294713630 0 0) 0x7f8a9c139180 con 0x7f8a9c13f710 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 >> v1:172.21.15.52:6790/0 conn(0x7f8a9c1430c0 legacy=0x7f8a9c1e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 >> v1:172.21.15.179:6789/0 conn(0x7f8a9c0a98f0 legacy=0x7f8a9c1dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8a83fff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8a83fff640 1 monclient: found mon.a 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8a83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8a83fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c137200 con 0x7f8a9c13f710 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8a83fff640 10 monclient: _finish_auth 0 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8a83fff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8a83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:24.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8aa1d2b640 5 monclient: authenticate success, global_id 4263 2024-03-25T19:28:24.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8aa1d2b640 10 monclient: _renew_subs 2024-03-25T19:28:24.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8aa1d2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:24.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/1237772950 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c0b7290 con 0x7f8a9c13f710 2024-03-25T19:28:24.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8aa1d2b640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:24.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8aa1d2b640 10 monclient: _renew_subs 2024-03-25T19:28:24.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8aa1d2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:24.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/1237772950 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c133680 con 0x7f8a9c13f710 2024-03-25T19:28:24.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.479+0000 7f8aa1d2b640 1 librados: init done 2024-03-25T19:28:24.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.480+0000 7f8aa1d2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:24.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.480+0000 7f8aa1d2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:24.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.480+0000 7f8aa1d2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:24.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.480+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/1237772950 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9c0627b0 con 0x7f8a9c13f710 2024-03-25T19:28:24.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.777+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a90001b90 con 0x7f8a9c13f710 2024-03-25T19:28:24.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.777+0000 7f8a83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:24.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.777+0000 7f8a99ffb640 4 set_mon_vals no callback set 2024-03-25T19:28:24.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.788+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8a90003a60 con 0x7f8a9c13f710 2024-03-25T19:28:24.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.788+0000 7f8a83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:24.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.788+0000 7f8a83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:24.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.788+0000 7f8a83fff640 10 monclient: dump: 2024-03-25T19:28:24.789 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:24.789 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:24.789 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:24.789 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:24.789 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:24.790 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:24.790 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:24.790 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:24.790 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:24.790 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:24.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.790+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f8a90003c40 con 0x7f8a9c13f710 2024-03-25T19:28:24.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.790+0000 7f8a9b7fe640 10 monclient: get_auth_request con 0x7f8a8806ddf0 auth_method 0 2024-03-25T19:28:24.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.790+0000 7f8a83fff640 10 client.4263.objecter ms_handle_connect 0x7f8a8806ddf0 2024-03-25T19:28:24.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.795+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f8a90054950 con 0x7f8a9c13f710 2024-03-25T19:28:24.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.795+0000 7f8a83fff640 10 client.4263.objecter ms_dispatch 0x7f8a9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:24.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.795+0000 7f8a83fff640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:24.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.795+0000 7f8a83fff640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:24.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.795+0000 7f8a83fff640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-25T19:28:24.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.795+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a90050290 con 0x7f8a9c13f710 2024-03-25T19:28:24.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.795+0000 7f8a83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:24.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:24.795+0000 7f8a83fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:25.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.051+0000 7f8aa1d2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:25.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.051+0000 7f8aa1d2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:25.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.051+0000 7f8aa1d2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:25.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.052+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/1237772950 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a9c13e2b0 con 0x7f8a9c13f710 2024-03-25T19:28:25.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.077+0000 7f8a83fff640 1 -- 172.21.15.52:0/1237772950 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f8a90052390 con 0x7f8a9c13f710 2024-03-25T19:28:25.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.077+0000 7f8a83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:25.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.077+0000 7f8a83fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:25.078 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:28:25.078 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:25.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 10 librados: watch_flush enter 2024-03-25T19:28:25.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 10 librados: watch_flush exit 2024-03-25T19:28:25.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-25T19:28:25.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 10 client.4263.objecter successfully canceled tick 2024-03-25T19:28:25.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/1237772950 >> v1:172.21.15.52:6800/3967427147 conn(0x7f8a8806ddf0 legacy=0x7f8a880702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 10 monclient: shutdown 2024-03-25T19:28:25.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:25.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/1237772950 >> v1:172.21.15.52:6789/0 conn(0x7f8a9c13f710 legacy=0x7f8a9c132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:25.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/1237772950 shutdown_connections 2024-03-25T19:28:25.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/1237772950 >> 172.21.15.52:0/1237772950 conn(0x7f8a9c0b7ea0 msgr2=0x7f8a9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:25.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/1237772950 shutdown_connections 2024-03-25T19:28:25.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.079+0000 7f8aa1d2b640 1 -- 172.21.15.52:0/1237772950 wait complete. 2024-03-25T19:28:25.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:25.080+0000 7f8aa1d2b640 1 librados: shutdown 2024-03-25T19:28:25.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:31.897 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:32.011 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.011+0000 7f84316d9640 10 monclient: get_monmap_and_config 2024-03-25T19:28:32.011 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.011+0000 7f84316d9640 10 monclient: build_initial_monmap 2024-03-25T19:28:32.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.011+0000 7f84316d9640 10 monclient: monmap: 2024-03-25T19:28:32.012 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:32.012 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:32.013 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:28:32.012658+0000 2024-03-25T19:28:32.013 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:28:32.012658+0000 2024-03-25T19:28:32.013 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:32.013 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:32.013 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:32.013 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:28:32.013 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:28:32.014 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.011+0000 7f84316d9640 1 Processor -- start 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.011+0000 7f84316d9640 1 -- start start 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.011+0000 7f84316d9640 10 monclient: init 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.012+0000 7f84316d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.012+0000 7f84316d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.012+0000 7f84316d9640 10 monclient(hunting): picked mon.noname-b con 0x7f842c0a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.012+0000 7f84316d9640 10 monclient(hunting): picked mon.noname-c con 0x7f842c13f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.012+0000 7f84316d9640 10 monclient(hunting): picked mon.noname-a con 0x7f842c1430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.012+0000 7f84316d9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842c142520 con 0x7f842c1430c0 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.012+0000 7f84316d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842c0b28a0 con 0x7f842c13f710 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.012+0000 7f84316d9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842c0b2a30 con 0x7f842c0a98f0 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.012+0000 7f84316d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:32.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.012+0000 7f84316d9640 10 monclient(hunting): authenticate will time out at 5960.193604s 2024-03-25T19:28:32.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.022+0000 7f842a7fc640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f842c13f710 0x7f842c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:60292/0 (socket says 172.21.15.52:60292) 2024-03-25T19:28:32.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.022+0000 7f842a7fc640 1 -- 172.21.15.52:0/3164312907 learned_addr learned my addr 172.21.15.52:0/3164312907 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:32.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.040+0000 7f8429ffb640 1 -- 172.21.15.52:0/3164312907 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428791832 0 0) 0x7f842c0b28a0 con 0x7f842c13f710 2024-03-25T19:28:32.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.040+0000 7f8429ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:32.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.040+0000 7f8429ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:32.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.040+0000 7f8429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:32.040 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.040+0000 7f8429ffb640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:28:32.041 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.040+0000 7f8429ffb640 1 -- 172.21.15.52:0/3164312907 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8414003520 con 0x7f842c13f710 2024-03-25T19:28:32.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.048+0000 7f8429ffb640 1 -- 172.21.15.52:0/3164312907 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340115080 0 0) 0x7f842c0b2a30 con 0x7f842c0a98f0 2024-03-25T19:28:32.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.048+0000 7f8429ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:32.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.048+0000 7f8429ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:32.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.048+0000 7f8429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:32.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.048+0000 7f8429ffb640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:28:32.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.048+0000 7f8429ffb640 1 -- 172.21.15.52:0/3164312907 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f842c0b28a0 con 0x7f842c0a98f0 2024-03-25T19:28:32.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 1 -- 172.21.15.52:0/3164312907 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2706575707 0 0) 0x7f842c0b28a0 con 0x7f842c0a98f0 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 1 -- 172.21.15.52:0/3164312907 >> v1:172.21.15.179:6789/0 conn(0x7f842c13f710 legacy=0x7f842c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 1 -- 172.21.15.52:0/3164312907 >> v1:172.21.15.52:6789/0 conn(0x7f842c1430c0 legacy=0x7f842c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 1 monclient: found mon.noname-b 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 1 -- 172.21.15.52:0/3164312907 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c0b44c0 con 0x7f842c0a98f0 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 1 -- 172.21.15.52:0/3164312907 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f841c001670 con 0x7f842c0a98f0 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f8429ffb640 10 monclient: dump: 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:32.062 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:32.063 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:32.063 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:32.063 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:32.063 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:32.063 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:32.063 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:32.063 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:32.063 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:32.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f84316d9640 5 monclient: authenticate success, global_id 4262 2024-03-25T19:28:32.063 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.061+0000 7f84316d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:32.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.086+0000 7f8429ffb640 1 -- 172.21.15.52:0/3164312907 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f841c0017d0 con 0x7f842c0a98f0 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.086+0000 7f8429ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.086+0000 7f8429ffb640 1 -- 172.21.15.52:0/3164312907 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f841c001be0 con 0x7f842c0a98f0 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.086+0000 7f8429ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.086+0000 7f8429ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.086+0000 7f8429ffb640 10 monclient: dump: 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:32.087 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:32.088 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:32.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.086+0000 7f84316d9640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:32.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.086+0000 7f84316d9640 4 set_mon_vals no callback set 2024-03-25T19:28:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.087+0000 7f84316d9640 10 monclient: shutdown 2024-03-25T19:28:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.087+0000 7f84316d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.087+0000 7f84316d9640 1 -- 172.21.15.52:0/3164312907 >> v1:172.21.15.52:6790/0 conn(0x7f842c0a98f0 legacy=0x7f842c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.087+0000 7f84316d9640 1 -- 172.21.15.52:0/3164312907 shutdown_connections 2024-03-25T19:28:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.087+0000 7f84316d9640 1 -- 172.21.15.52:0/3164312907 >> 172.21.15.52:0/3164312907 conn(0x7f842c0b7ea0 msgr2=0x7f842c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.087+0000 7f84316d9640 1 -- 172.21.15.52:0/3164312907 shutdown_connections 2024-03-25T19:28:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.087+0000 7f84316d9640 1 -- 172.21.15.52:0/3164312907 wait complete. 2024-03-25T19:28:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.088+0000 7f84316d9640 10 monclient: build_initial_monmap 2024-03-25T19:28:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.088+0000 7f84316d9640 10 monclient: monmap: 2024-03-25T19:28:32.090 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.088+0000 7f84316d9640 1 librados: starting msgr at 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.088+0000 7f84316d9640 1 librados: starting objecter 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.088+0000 7f84316d9640 1 Processor -- start 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.088+0000 7f84316d9640 1 -- start start 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.088+0000 7f84316d9640 1 librados: setting wanted keys 2024-03-25T19:28:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.088+0000 7f84316d9640 1 librados: calling monclient init 2024-03-25T19:28:32.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.088+0000 7f84316d9640 10 monclient: init 2024-03-25T19:28:32.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.089+0000 7f84316d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:32.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.089+0000 7f84316d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:32.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.089+0000 7f84316d9640 10 monclient(hunting): picked mon.noname-b con 0x7f842c0a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:32.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.089+0000 7f84316d9640 10 monclient(hunting): picked mon.noname-a con 0x7f842c13f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:32.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.089+0000 7f84316d9640 10 monclient(hunting): picked mon.noname-c con 0x7f842c1430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:32.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.089+0000 7f84316d9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842c1d7750 con 0x7f842c13f710 2024-03-25T19:28:32.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.089+0000 7f84316d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842c136750 con 0x7f842c0a98f0 2024-03-25T19:28:32.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.089+0000 7f84316d9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842c1d85d0 con 0x7f842c1430c0 2024-03-25T19:28:32.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.089+0000 7f84316d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:32.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.089+0000 7f84316d9640 10 monclient(hunting): authenticate will time out at 5960.270591s 2024-03-25T19:28:32.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.097+0000 7f842affd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f842c0a98f0 0x7f842c1d7ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:60370/0 (socket says 172.21.15.52:60370) 2024-03-25T19:28:32.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.097+0000 7f842affd640 1 -- 172.21.15.52:0/613229813 learned_addr learned my addr 172.21.15.52:0/613229813 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:32.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.100+0000 7f84137fe640 10 client.?.objecter ms_handle_connect 0x7f842c0a98f0 2024-03-25T19:28:32.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.100+0000 7f84137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:32.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.105+0000 7f84137fe640 10 client.?.objecter ms_handle_connect 0x7f842c1430c0 2024-03-25T19:28:32.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.105+0000 7f84137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:32.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.105+0000 7f84137fe640 10 client.?.objecter ms_handle_connect 0x7f842c13f710 2024-03-25T19:28:32.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.105+0000 7f84137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.115+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050289784 0 0) 0x7f842c1d7750 con 0x7f842c13f710 2024-03-25T19:28:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.115+0000 7f84137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.115+0000 7f84137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.115+0000 7f84137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.115+0000 7f84137fe640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:28:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.115+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8418004eb0 con 0x7f842c13f710 2024-03-25T19:28:32.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.118+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978880671 0 0) 0x7f842c1d85d0 con 0x7f842c1430c0 2024-03-25T19:28:32.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.118+0000 7f84137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:32.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.118+0000 7f84137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:32.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.118+0000 7f84137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:32.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.118+0000 7f84137fe640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:28:32.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.118+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f842c1d7750 con 0x7f842c1430c0 2024-03-25T19:28:32.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.124+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952763642 0 0) 0x7f842c136750 con 0x7f842c0a98f0 2024-03-25T19:28:32.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.124+0000 7f84137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:32.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.124+0000 7f84137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:32.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.124+0000 7f84137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:32.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.124+0000 7f84137fe640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:28:32.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.124+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f842c1d85d0 con 0x7f842c0a98f0 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.125+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 981562028 0 0) 0x7f8418004eb0 con 0x7f842c13f710 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.125+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f842c136750 con 0x7f842c13f710 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.125+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8420001950 con 0x7f842c13f710 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.125+0000 7f84137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.125+0000 7f84137fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.125+0000 7f84137fe640 10 monclient(hunting): dump: 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:32.126 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:32.127 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:32.127 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:32.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.125+0000 7f84137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:32.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3044540003 0 0) 0x7f842c1d7750 con 0x7f842c1430c0 2024-03-25T19:28:32.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8418004eb0 con 0x7f842c1430c0 2024-03-25T19:28:32.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8424001b10 con 0x7f842c1430c0 2024-03-25T19:28:32.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:32.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:32.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 10 monclient(hunting): dump: 2024-03-25T19:28:32.135 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:32.135 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:32.135 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:32.135 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:32.135 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:32.135 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:32.135 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:32.135 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:32.135 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:32.135 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:32.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1796204435 0 0) 0x7f842c136750 con 0x7f842c13f710 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 >> v1:172.21.15.52:6790/0 conn(0x7f842c1430c0 legacy=0x7f842c1d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 >> v1:172.21.15.179:6789/0 conn(0x7f842c0a98f0 legacy=0x7f842c1d7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 1 monclient: found mon.a 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c0ab950 con 0x7f842c13f710 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84316d9640 5 monclient: authenticate success, global_id 4266 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84316d9640 10 monclient: _renew_subs 2024-03-25T19:28:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84316d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:32.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.134+0000 7f84316d9640 1 -- 172.21.15.52:0/613229813 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842c1d7a00 con 0x7f842c13f710 2024-03-25T19:28:32.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.135+0000 7f84316d9640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:32.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.135+0000 7f84316d9640 10 monclient: _renew_subs 2024-03-25T19:28:32.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.135+0000 7f84316d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:32.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.135+0000 7f84316d9640 1 -- 172.21.15.52:0/613229813 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f842c138e60 con 0x7f842c13f710 2024-03-25T19:28:32.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.135+0000 7f84316d9640 1 librados: init done 2024-03-25T19:28:32.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.136+0000 7f84316d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:32.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.136+0000 7f84316d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:32.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.136+0000 7f84316d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:32.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.136+0000 7f84316d9640 1 -- 172.21.15.52:0/613229813 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842c0627b0 con 0x7f842c13f710 2024-03-25T19:28:32.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.456+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8420001d60 con 0x7f842c13f710 2024-03-25T19:28:32.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.456+0000 7f84137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:32.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.456+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8420004980 con 0x7f842c13f710 2024-03-25T19:28:32.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.456+0000 7f84137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:32.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.456+0000 7f84137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:32.457 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.456+0000 7f84137fe640 10 monclient: dump: 2024-03-25T19:28:32.457 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:32.457 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:32.457 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:32.458 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:32.458 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:32.458 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:32.458 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:32.458 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:32.458 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:32.458 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:32.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.456+0000 7f84297fa640 4 set_mon_vals no callback set 2024-03-25T19:28:32.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.457+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f842001a9d0 con 0x7f842c13f710 2024-03-25T19:28:32.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.458+0000 7f842affd640 10 monclient: get_auth_request con 0x7f841806df10 auth_method 0 2024-03-25T19:28:32.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.458+0000 7f84137fe640 10 client.4266.objecter ms_handle_connect 0x7f841806df10 2024-03-25T19:28:32.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.467+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f84200546f0 con 0x7f842c13f710 2024-03-25T19:28:32.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.467+0000 7f84137fe640 10 client.4266.objecter ms_dispatch 0x7f842c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:32.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.467+0000 7f84137fe640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:32.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.467+0000 7f84137fe640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:32.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.467+0000 7f84137fe640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-25T19:28:32.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.467+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8420050030 con 0x7f842c13f710 2024-03-25T19:28:32.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.467+0000 7f84137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:32.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.467+0000 7f84137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:32.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.700+0000 7f84316d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:32.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.700+0000 7f84316d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:32.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.700+0000 7f84316d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:32.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.700+0000 7f84316d9640 1 -- 172.21.15.52:0/613229813 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f842c074cc0 con 0x7f842c13f710 2024-03-25T19:28:32.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.722+0000 7f84137fe640 1 -- 172.21.15.52:0/613229813 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f8420052130 con 0x7f842c13f710 2024-03-25T19:28:32.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.722+0000 7f84137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:32.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.722+0000 7f84137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:32.723 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:28:32.723 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:32.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.724+0000 7f84316d9640 10 librados: watch_flush enter 2024-03-25T19:28:32.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.724+0000 7f84316d9640 10 librados: watch_flush exit 2024-03-25T19:28:32.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.724+0000 7f84316d9640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-25T19:28:32.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.724+0000 7f84316d9640 10 client.4266.objecter successfully canceled tick 2024-03-25T19:28:32.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.724+0000 7f84316d9640 1 -- 172.21.15.52:0/613229813 >> v1:172.21.15.52:6800/3967427147 conn(0x7f841806df10 legacy=0x7f84180703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.724+0000 7f84316d9640 10 monclient: shutdown 2024-03-25T19:28:32.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.724+0000 7f84316d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:32.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.724+0000 7f84316d9640 1 -- 172.21.15.52:0/613229813 >> v1:172.21.15.52:6789/0 conn(0x7f842c13f710 legacy=0x7f842c138300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:32.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.725+0000 7f84316d9640 1 -- 172.21.15.52:0/613229813 shutdown_connections 2024-03-25T19:28:32.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.725+0000 7f84316d9640 1 -- 172.21.15.52:0/613229813 >> 172.21.15.52:0/613229813 conn(0x7f842c0b7ea0 msgr2=0x7f842c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:32.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.725+0000 7f84316d9640 1 -- 172.21.15.52:0/613229813 shutdown_connections 2024-03-25T19:28:32.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.725+0000 7f84316d9640 1 -- 172.21.15.52:0/613229813 wait complete. 2024-03-25T19:28:32.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:32.725+0000 7f84316d9640 1 librados: shutdown 2024-03-25T19:28:32.737 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:39.543 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:39.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.663+0000 7f46fee06640 10 monclient: get_monmap_and_config 2024-03-25T19:28:39.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.663+0000 7f46fee06640 10 monclient: build_initial_monmap 2024-03-25T19:28:39.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.663+0000 7f46fee06640 10 monclient: monmap: 2024-03-25T19:28:39.664 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:39.664 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:39.664 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:28:39.664747+0000 2024-03-25T19:28:39.664 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:28:39.664747+0000 2024-03-25T19:28:39.664 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:39.664 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:39.664 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:39.664 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:28:39.664 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:28:39.664 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:39.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.664+0000 7f46fee06640 1 Processor -- start 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.664+0000 7f46fee06640 1 -- start start 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.664+0000 7f46fee06640 10 monclient: init 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.664+0000 7f46fee06640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.664+0000 7f46fee06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.664+0000 7f46fee06640 10 monclient(hunting): picked mon.noname-b con 0x7f46f80a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.664+0000 7f46fee06640 10 monclient(hunting): picked mon.noname-c con 0x7f46f813f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.665+0000 7f46fee06640 10 monclient(hunting): picked mon.noname-a con 0x7f46f81430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.665+0000 7f46fee06640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f8142520 con 0x7f46f81430c0 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.665+0000 7f46fee06640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f80b28a0 con 0x7f46f813f710 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.665+0000 7f46fee06640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f80b2a30 con 0x7f46f80a98f0 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.665+0000 7f46fee06640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:39.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.665+0000 7f46fee06640 10 monclient(hunting): authenticate will time out at 5967.846078s 2024-03-25T19:28:39.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.682+0000 7f46fcba2640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f46f80a98f0 0x7f46f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:53618/0 (socket says 172.21.15.52:53618) 2024-03-25T19:28:39.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.682+0000 7f46fcba2640 1 -- 172.21.15.52:0/754523301 learned_addr learned my addr 172.21.15.52:0/754523301 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:39.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.689+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913677270 0 0) 0x7f46f80b28a0 con 0x7f46f813f710 2024-03-25T19:28:39.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.689+0000 7f46f77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.689+0000 7f46f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.689+0000 7f46f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.689+0000 7f46f77fe640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:28:39.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.689+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e8003520 con 0x7f46f813f710 2024-03-25T19:28:39.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.700+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711780904 0 0) 0x7f46f80b2a30 con 0x7f46f80a98f0 2024-03-25T19:28:39.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.700+0000 7f46f77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.700+0000 7f46f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.700+0000 7f46f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.700+0000 7f46f77fe640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:28:39.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.700+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f80b28a0 con 0x7f46f80a98f0 2024-03-25T19:28:39.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.708+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203325843 0 0) 0x7f46f8142520 con 0x7f46f81430c0 2024-03-25T19:28:39.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.708+0000 7f46f77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.708+0000 7f46f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.708+0000 7f46f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.708+0000 7f46f77fe640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:28:39.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.708+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f80b2a30 con 0x7f46f81430c0 2024-03-25T19:28:39.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4122157150 0 0) 0x7f46f80b28a0 con 0x7f46f80a98f0 2024-03-25T19:28:39.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 >> v1:172.21.15.179:6789/0 conn(0x7f46f813f710 legacy=0x7f46f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 >> v1:172.21.15.52:6789/0 conn(0x7f46f81430c0 legacy=0x7f46f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:39.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 1 monclient: found mon.noname-b 2024-03-25T19:28:39.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:39.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f80b44c0 con 0x7f46f80a98f0 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46ec001670 con 0x7f46f80a98f0 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46f77fe640 10 monclient: dump: 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:39.718 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:39.719 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:39.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46fee06640 5 monclient: authenticate success, global_id 4268 2024-03-25T19:28:39.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.716+0000 7f46fee06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:39.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.734+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46ec0017d0 con 0x7f46f80a98f0 2024-03-25T19:28:39.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.734+0000 7f46f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:39.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.734+0000 7f46f77fe640 1 -- 172.21.15.52:0/754523301 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46ec001be0 con 0x7f46f80a98f0 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.734+0000 7f46f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.734+0000 7f46f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.734+0000 7f46f77fe640 10 monclient: dump: 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:39.735 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:39.736 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:39.736 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.734+0000 7f46fee06640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:39.736 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.734+0000 7f46fee06640 4 set_mon_vals no callback set 2024-03-25T19:28:39.737 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.735+0000 7f46fee06640 10 monclient: shutdown 2024-03-25T19:28:39.737 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.735+0000 7f46fee06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:39.737 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.735+0000 7f46fee06640 1 -- 172.21.15.52:0/754523301 >> v1:172.21.15.52:6790/0 conn(0x7f46f80a98f0 legacy=0x7f46f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.737 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.735+0000 7f46fee06640 1 -- 172.21.15.52:0/754523301 shutdown_connections 2024-03-25T19:28:39.737 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.735+0000 7f46fee06640 1 -- 172.21.15.52:0/754523301 >> 172.21.15.52:0/754523301 conn(0x7f46f80b7ea0 msgr2=0x7f46f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.735+0000 7f46fee06640 1 -- 172.21.15.52:0/754523301 shutdown_connections 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.735+0000 7f46fee06640 1 -- 172.21.15.52:0/754523301 wait complete. 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.736+0000 7f46fee06640 10 monclient: build_initial_monmap 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.736+0000 7f46fee06640 10 monclient: monmap: 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:28:39.738 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:39.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.736+0000 7f46fee06640 1 librados: starting msgr at 2024-03-25T19:28:39.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.736+0000 7f46fee06640 1 librados: starting objecter 2024-03-25T19:28:39.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.736+0000 7f46fee06640 1 Processor -- start 2024-03-25T19:28:39.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.736+0000 7f46fee06640 1 -- start start 2024-03-25T19:28:39.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.736+0000 7f46fee06640 1 librados: setting wanted keys 2024-03-25T19:28:39.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.736+0000 7f46fee06640 1 librados: calling monclient init 2024-03-25T19:28:39.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.736+0000 7f46fee06640 10 monclient: init 2024-03-25T19:28:39.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.737+0000 7f46fee06640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:39.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.737+0000 7f46fee06640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:39.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.737+0000 7f46fee06640 10 monclient(hunting): picked mon.noname-b con 0x7f46f80a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:39.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.737+0000 7f46fee06640 10 monclient(hunting): picked mon.noname-c con 0x7f46f813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:39.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.737+0000 7f46fee06640 10 monclient(hunting): picked mon.noname-a con 0x7f46f81430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:39.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.737+0000 7f46fee06640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f8136e60 con 0x7f46f81430c0 2024-03-25T19:28:39.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.737+0000 7f46fee06640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f81381a0 con 0x7f46f80a98f0 2024-03-25T19:28:39.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.737+0000 7f46fee06640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f81370b0 con 0x7f46f813f710 2024-03-25T19:28:39.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.738+0000 7f46fee06640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:39.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.738+0000 7f46fee06640 10 monclient(hunting): authenticate will time out at 5967.918924s 2024-03-25T19:28:39.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.742+0000 7f46fcba2640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f46f80a98f0 0x7f46f8137a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:51482/0 (socket says 172.21.15.52:51482) 2024-03-25T19:28:39.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.742+0000 7f46fcba2640 1 -- 172.21.15.52:0/333541012 learned_addr learned my addr 172.21.15.52:0/333541012 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:39.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.744+0000 7f46f57fa640 10 client.?.objecter ms_handle_connect 0x7f46f80a98f0 2024-03-25T19:28:39.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.745+0000 7f46f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:39.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.745+0000 7f46f57fa640 10 client.?.objecter ms_handle_connect 0x7f46f81430c0 2024-03-25T19:28:39.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.745+0000 7f46f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:39.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.756+0000 7f46f57fa640 10 client.?.objecter ms_handle_connect 0x7f46f813f710 2024-03-25T19:28:39.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.756+0000 7f46f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:39.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.762+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077174325 0 0) 0x7f46f8136e60 con 0x7f46f81430c0 2024-03-25T19:28:39.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.762+0000 7f46f57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.762+0000 7f46f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.762+0000 7f46f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.762+0000 7f46f57fa640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:28:39.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.762+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc004eb0 con 0x7f46f81430c0 2024-03-25T19:28:39.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.763+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071780276 0 0) 0x7f46f81381a0 con 0x7f46f80a98f0 2024-03-25T19:28:39.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.763+0000 7f46f57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.763+0000 7f46f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.763+0000 7f46f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.763+0000 7f46f57fa640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:28:39.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.763+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f8136e60 con 0x7f46f80a98f0 2024-03-25T19:28:39.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.766+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779200835 0 0) 0x7f46f81370b0 con 0x7f46f813f710 2024-03-25T19:28:39.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.766+0000 7f46f57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:39.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.766+0000 7f46f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:39.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.766+0000 7f46f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:39.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.766+0000 7f46f57fa640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:28:39.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.766+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f81381a0 con 0x7f46f813f710 2024-03-25T19:28:39.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.775+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 906938522 0 0) 0x7f46f8136e60 con 0x7f46f80a98f0 2024-03-25T19:28:39.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.775+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46f81370b0 con 0x7f46f80a98f0 2024-03-25T19:28:39.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.775+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46ec001ba0 con 0x7f46f80a98f0 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.775+0000 7f46f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.775+0000 7f46f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.775+0000 7f46f57fa640 10 monclient(hunting): dump: 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:39.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.775+0000 7f46f57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:39.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.778+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 428553813 0 0) 0x7f46dc004eb0 con 0x7f46f81430c0 2024-03-25T19:28:39.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.778+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46f8136e60 con 0x7f46f81430c0 2024-03-25T19:28:39.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.778+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46f0001bd0 con 0x7f46f81430c0 2024-03-25T19:28:39.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.778+0000 7f46f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:39.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.778+0000 7f46f57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:39.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.778+0000 7f46f57fa640 10 monclient(hunting): dump: 2024-03-25T19:28:39.781 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:39.781 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:39.781 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:39.782 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:39.782 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:39.782 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:39.782 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:39.782 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:39.782 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:39.782 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:39.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.780+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439862138 0 0) 0x7f46f81381a0 con 0x7f46f813f710 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.780+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46dc004eb0 con 0x7f46f813f710 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.780+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46e40017b0 con 0x7f46f813f710 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.780+0000 7f46f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.780+0000 7f46f57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.780+0000 7f46f57fa640 10 monclient(hunting): dump: 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:39.784 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3637692233 0 0) 0x7f46f81370b0 con 0x7f46f80a98f0 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 >> v1:172.21.15.52:6790/0 conn(0x7f46f813f710 legacy=0x7f46f813a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 >> v1:172.21.15.52:6789/0 conn(0x7f46f81430c0 legacy=0x7f46f813ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46f57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46f57fa640 1 monclient: found mon.b 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46f57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f8137390 con 0x7f46f80a98f0 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46f57fa640 10 monclient: _finish_auth 0 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46f57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46fee06640 5 monclient: authenticate success, global_id 4279 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46fee06640 10 monclient: _renew_subs 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46fee06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:28:39.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46fee06640 1 -- 172.21.15.52:0/333541012 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f81376d0 con 0x7f46f80a98f0 2024-03-25T19:28:39.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46fee06640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:39.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46fee06640 10 monclient: _renew_subs 2024-03-25T19:28:39.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46fee06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:28:39.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46fee06640 1 -- 172.21.15.52:0/333541012 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f8135530 con 0x7f46f80a98f0 2024-03-25T19:28:39.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.782+0000 7f46fee06640 1 librados: init done 2024-03-25T19:28:39.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.783+0000 7f46fee06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:39.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.783+0000 7f46fee06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:39.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.783+0000 7f46fee06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:28:39.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:39.783+0000 7f46fee06640 1 -- 172.21.15.52:0/333541012 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f80b3ef0 con 0x7f46f80a98f0 2024-03-25T19:28:40.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.069+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46ec004a90 con 0x7f46f80a98f0 2024-03-25T19:28:40.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.069+0000 7f46f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:40.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.069+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46ec004da0 con 0x7f46f80a98f0 2024-03-25T19:28:40.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.069+0000 7f46f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:40.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.069+0000 7f46f6ffd640 4 set_mon_vals no callback set 2024-03-25T19:28:40.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.069+0000 7f46f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:40.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.069+0000 7f46f57fa640 10 monclient: dump: 2024-03-25T19:28:40.070 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:40.070 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:40.071 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:40.071 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:40.071 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:40.071 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:40.071 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:40.071 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:40.071 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:40.071 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:40.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.069+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f46ec01ad70 con 0x7f46f80a98f0 2024-03-25T19:28:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.070+0000 7f46f7fff640 10 monclient: get_auth_request con 0x7f46dc06e0f0 auth_method 0 2024-03-25T19:28:40.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.070+0000 7f46f57fa640 10 client.4279.objecter ms_handle_connect 0x7f46dc06e0f0 2024-03-25T19:28:40.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.075+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f46ec054930 con 0x7f46f80a98f0 2024-03-25T19:28:40.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.075+0000 7f46f57fa640 10 client.4279.objecter ms_dispatch 0x7f46f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:40.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.075+0000 7f46f57fa640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:40.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.075+0000 7f46f57fa640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:40.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.075+0000 7f46f57fa640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-25T19:28:40.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.075+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46ec050270 con 0x7f46f80a98f0 2024-03-25T19:28:40.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.075+0000 7f46f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:40.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.075+0000 7f46f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:40.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.307+0000 7f46fee06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:40.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.307+0000 7f46fee06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:40.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.307+0000 7f46fee06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:28:40.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.307+0000 7f46fee06640 1 -- 172.21.15.52:0/333541012 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46f81357e0 con 0x7f46f80a98f0 2024-03-25T19:28:40.346 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.346+0000 7f46f57fa640 1 -- 172.21.15.52:0/333541012 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f46ec052370 con 0x7f46f80a98f0 2024-03-25T19:28:40.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.346+0000 7f46f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:40.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.346+0000 7f46f57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:40.347 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:28:40.347 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:40.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 10 librados: watch_flush enter 2024-03-25T19:28:40.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 10 librados: watch_flush exit 2024-03-25T19:28:40.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-25T19:28:40.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 10 client.4279.objecter successfully canceled tick 2024-03-25T19:28:40.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 1 -- 172.21.15.52:0/333541012 >> v1:172.21.15.52:6800/3967427147 conn(0x7f46dc06e0f0 legacy=0x7f46dc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:40.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 10 monclient: shutdown 2024-03-25T19:28:40.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:40.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 1 -- 172.21.15.52:0/333541012 >> v1:172.21.15.179:6789/0 conn(0x7f46f80a98f0 legacy=0x7f46f8137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:40.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 1 -- 172.21.15.52:0/333541012 shutdown_connections 2024-03-25T19:28:40.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 1 -- 172.21.15.52:0/333541012 >> 172.21.15.52:0/333541012 conn(0x7f46f80b7ea0 msgr2=0x7f46f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:40.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 1 -- 172.21.15.52:0/333541012 shutdown_connections 2024-03-25T19:28:40.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 1 -- 172.21.15.52:0/333541012 wait complete. 2024-03-25T19:28:40.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:40.348+0000 7f46fee06640 1 librados: shutdown 2024-03-25T19:28:40.361 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:40.894 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:01:34.493 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:40.949 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:01:34.548 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:41.005 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:01:34.546 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:41.007 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:01:34.606 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:41.009 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:01:34.551 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:41.013 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:01:34.610 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:41.014 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:01:34.611 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:41.051 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:01:34.593 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:41.075 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:01:34.674 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:47.168 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:47.282 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.282+0000 7f7ff0154640 10 monclient: get_monmap_and_config 2024-03-25T19:28:47.282 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.282+0000 7f7ff0154640 10 monclient: build_initial_monmap 2024-03-25T19:28:47.282 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.282+0000 7f7ff0154640 10 monclient: monmap: 2024-03-25T19:28:47.282 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:47.282 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:47.282 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:28:47.283339+0000 2024-03-25T19:28:47.282 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:28:47.283339+0000 2024-03-25T19:28:47.283 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:47.283 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:47.283 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:47.283 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:28:47.283 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:28:47.283 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.283+0000 7f7ff0154640 1 Processor -- start 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.283+0000 7f7ff0154640 1 -- start start 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.283+0000 7f7ff0154640 10 monclient: init 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.284+0000 7f7ff0154640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.284+0000 7f7ff0154640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.284+0000 7f7ff0154640 10 monclient(hunting): picked mon.noname-b con 0x7f7fe8061890 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.284+0000 7f7ff0154640 10 monclient(hunting): picked mon.noname-a con 0x7f7fe813f520 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:47.284 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.284+0000 7f7ff0154640 10 monclient(hunting): picked mon.noname-c con 0x7f7fe813f9a0 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:47.285 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.284+0000 7f7ff0154640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe806fe30 con 0x7f7fe813f520 2024-03-25T19:28:47.285 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.284+0000 7f7ff0154640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe80ae5b0 con 0x7f7fe813f9a0 2024-03-25T19:28:47.285 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.284+0000 7f7ff0154640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe813fdd0 con 0x7f7fe8061890 2024-03-25T19:28:47.285 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.284+0000 7f7ff0154640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:47.285 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.284+0000 7f7ff0154640 10 monclient(hunting): authenticate will time out at 5975.464508s 2024-03-25T19:28:47.290 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.290+0000 7f7fee6f1640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f7fe813f9a0 0x7f7fe8146120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:43308/0 (socket says 172.21.15.52:43308) 2024-03-25T19:28:47.290 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.290+0000 7f7fee6f1640 1 -- 172.21.15.52:0/954856068 learned_addr learned my addr 172.21.15.52:0/954856068 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:47.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.300+0000 7f7feceee640 1 -- 172.21.15.52:0/954856068 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111460665 0 0) 0x7f7fe80ae5b0 con 0x7f7fe813f9a0 2024-03-25T19:28:47.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.300+0000 7f7feceee640 10 monclient(hunting): _negotiate 2024-03-25T19:28:47.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.300+0000 7f7feceee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:47.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.300+0000 7f7feceee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:47.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.300+0000 7f7feceee640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:28:47.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.300+0000 7f7feceee640 1 -- 172.21.15.52:0/954856068 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd4003520 con 0x7f7fe813f9a0 2024-03-25T19:28:47.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 1 -- 172.21.15.52:0/954856068 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4111028611 0 0) 0x7f7fd4003520 con 0x7f7fe813f9a0 2024-03-25T19:28:47.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 1 -- 172.21.15.52:0/954856068 >> v1:172.21.15.52:6790/0 conn(0x7f7fe8061890 legacy=0x7f7fe80adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:28:47.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 1 -- 172.21.15.52:0/954856068 >> v1:172.21.15.52:6789/0 conn(0x7f7fe813f520 legacy=0x7f7fe8145a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 10 monclient: _finish_hunting 0 2024-03-25T19:28:47.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 1 monclient: found mon.noname-c 2024-03-25T19:28:47.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:47.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:28:47.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 1 -- 172.21.15.52:0/954856068 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe8151bd0 con 0x7f7fe813f9a0 2024-03-25T19:28:47.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 10 monclient: _finish_auth 0 2024-03-25T19:28:47.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 10 monclient: _check_auth_tickets 2024-03-25T19:28:47.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7ff0154640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 1 -- 172.21.15.52:0/954856068 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7fe0001a10 con 0x7f7fe813f9a0 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7feceee640 10 monclient: dump: 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:47.312 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:47.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.311+0000 7f7ff0154640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:47.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.320+0000 7f7feceee640 1 -- 172.21.15.52:0/954856068 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fe0001b70 con 0x7f7fe813f9a0 2024-03-25T19:28:47.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.320+0000 7f7feceee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:47.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.320+0000 7f7feceee640 1 -- 172.21.15.52:0/954856068 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7fe0001fc0 con 0x7f7fe813f9a0 2024-03-25T19:28:47.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.320+0000 7f7feceee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.320+0000 7f7feceee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.320+0000 7f7feceee640 10 monclient: dump: 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.320+0000 7f7ff0154640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:47.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.320+0000 7f7ff0154640 4 set_mon_vals no callback set 2024-03-25T19:28:47.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.321+0000 7f7ff0154640 10 monclient: shutdown 2024-03-25T19:28:47.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.321+0000 7f7ff0154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:47.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.321+0000 7f7ff0154640 1 -- 172.21.15.52:0/954856068 >> v1:172.21.15.179:6789/0 conn(0x7f7fe813f9a0 legacy=0x7f7fe8146120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.321+0000 7f7ff0154640 1 -- 172.21.15.52:0/954856068 shutdown_connections 2024-03-25T19:28:47.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.321+0000 7f7ff0154640 1 -- 172.21.15.52:0/954856068 >> 172.21.15.52:0/954856068 conn(0x7f7fe80b0cc0 msgr2=0x7f7fe80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:47.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.321+0000 7f7ff0154640 1 -- 172.21.15.52:0/954856068 shutdown_connections 2024-03-25T19:28:47.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.321+0000 7f7ff0154640 1 -- 172.21.15.52:0/954856068 wait complete. 2024-03-25T19:28:47.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.322+0000 7f7ff0154640 10 monclient: build_initial_monmap 2024-03-25T19:28:47.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.322+0000 7f7ff0154640 10 monclient: monmap: 2024-03-25T19:28:47.322 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.322+0000 7f7ff0154640 1 librados: starting msgr at 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.322+0000 7f7ff0154640 1 librados: starting objecter 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.322+0000 7f7ff0154640 1 Processor -- start 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.322+0000 7f7ff0154640 1 -- start start 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.322+0000 7f7ff0154640 1 librados: setting wanted keys 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.322+0000 7f7ff0154640 1 librados: calling monclient init 2024-03-25T19:28:47.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.322+0000 7f7ff0154640 10 monclient: init 2024-03-25T19:28:47.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.323+0000 7f7ff0154640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:47.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.323+0000 7f7ff0154640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:47.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.323+0000 7f7ff0154640 10 monclient(hunting): picked mon.noname-a con 0x7f7fe8061890 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:47.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.323+0000 7f7ff0154640 10 monclient(hunting): picked mon.noname-b con 0x7f7fe813f520 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:47.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.323+0000 7f7ff0154640 10 monclient(hunting): picked mon.noname-c con 0x7f7fe813f9a0 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:47.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.323+0000 7f7ff0154640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe8141340 con 0x7f7fe8061890 2024-03-25T19:28:47.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.323+0000 7f7ff0154640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe8143020 con 0x7f7fe813f520 2024-03-25T19:28:47.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.323+0000 7f7ff0154640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe8140030 con 0x7f7fe813f9a0 2024-03-25T19:28:47.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.323+0000 7f7ff0154640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:47.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.323+0000 7f7ff0154640 10 monclient(hunting): authenticate will time out at 5975.503760s 2024-03-25T19:28:47.341 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.341+0000 7f7fed6ef640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f7fe813f520 0x7f7fe81db110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:43344/0 (socket says 172.21.15.52:43344) 2024-03-25T19:28:47.341 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.341+0000 7f7fed6ef640 1 -- 172.21.15.52:0/2384940924 learned_addr learned my addr 172.21.15.52:0/2384940924 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:47.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.349+0000 7f7fde7fc640 10 client.?.objecter ms_handle_connect 0x7f7fe813f520 2024-03-25T19:28:47.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.349+0000 7f7fde7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:47.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.350+0000 7f7fde7fc640 10 client.?.objecter ms_handle_connect 0x7f7fe813f9a0 2024-03-25T19:28:47.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.350+0000 7f7fde7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:47.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.356+0000 7f7fde7fc640 10 client.?.objecter ms_handle_connect 0x7f7fe8061890 2024-03-25T19:28:47.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.356+0000 7f7fde7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:47.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.358+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931282417 0 0) 0x7f7fe8143020 con 0x7f7fe813f520 2024-03-25T19:28:47.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.358+0000 7f7fde7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:47.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.358+0000 7f7fde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:47.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.358+0000 7f7fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:47.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.358+0000 7f7fde7fc640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:28:47.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.358+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fcc004eb0 con 0x7f7fe813f520 2024-03-25T19:28:47.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.365+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206217259 0 0) 0x7f7fe8141340 con 0x7f7fe8061890 2024-03-25T19:28:47.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.365+0000 7f7fde7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:47.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.365+0000 7f7fde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:47.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.365+0000 7f7fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:47.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.365+0000 7f7fde7fc640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:28:47.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.365+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe8143020 con 0x7f7fe8061890 2024-03-25T19:28:47.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1393698692 0 0) 0x7f7fcc004eb0 con 0x7f7fe813f520 2024-03-25T19:28:47.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fe8141340 con 0x7f7fe813f520 2024-03-25T19:28:47.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7fd8001a30 con 0x7f7fe813f520 2024-03-25T19:28:47.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:47.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:47.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 10 monclient(hunting): dump: 2024-03-25T19:28:47.369 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:47.369 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:47.369 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:47.369 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:47.369 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:47.369 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:47.369 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:47.369 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:47.369 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:47.369 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:47.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:47.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977452428 0 0) 0x7f7fe8140030 con 0x7f7fe813f9a0 2024-03-25T19:28:47.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:47.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:47.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:47.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:28:47.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.369+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fcc004eb0 con 0x7f7fe813f9a0 2024-03-25T19:28:47.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3584877859 0 0) 0x7f7fe8141340 con 0x7f7fe813f520 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 >> v1:172.21.15.52:6790/0 conn(0x7f7fe813f9a0 legacy=0x7f7fe81e2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 >> v1:172.21.15.52:6789/0 conn(0x7f7fe8061890 legacy=0x7f7fe8140c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7fde7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7fde7fc640 1 monclient: found mon.b 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7fde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7fde7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe8141e90 con 0x7f7fe813f520 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7fde7fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7fde7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7fde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7ff0154640 5 monclient: authenticate success, global_id 4285 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7ff0154640 10 monclient: _renew_subs 2024-03-25T19:28:47.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7ff0154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:28:47.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7ff0154640 1 -- 172.21.15.52:0/2384940924 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe8141a00 con 0x7f7fe813f520 2024-03-25T19:28:47.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7ff0154640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:47.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7ff0154640 10 monclient: _renew_subs 2024-03-25T19:28:47.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7ff0154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:28:47.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7ff0154640 1 -- 172.21.15.52:0/2384940924 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe81dba00 con 0x7f7fe813f520 2024-03-25T19:28:47.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.381+0000 7f7ff0154640 1 librados: init done 2024-03-25T19:28:47.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.382+0000 7f7ff0154640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:47.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.382+0000 7f7ff0154640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:47.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.382+0000 7f7ff0154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:28:47.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.382+0000 7f7ff0154640 1 -- 172.21.15.52:0/2384940924 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe81dbd00 con 0x7f7fe813f520 2024-03-25T19:28:47.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.664+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fd8001c70 con 0x7f7fe813f520 2024-03-25T19:28:47.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.664+0000 7f7fde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:47.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.664+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7fd8003a60 con 0x7f7fe813f520 2024-03-25T19:28:47.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.664+0000 7f7fde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.664+0000 7f7fde7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.664+0000 7f7fde7fc640 10 monclient: dump: 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.664+0000 7f7fdffff640 4 set_mon_vals no callback set 2024-03-25T19:28:47.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.665+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f7fd8003c40 con 0x7f7fe813f520 2024-03-25T19:28:47.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.665+0000 7f7fedef0640 10 monclient: get_auth_request con 0x7f7fcc06de00 auth_method 0 2024-03-25T19:28:47.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.666+0000 7f7fde7fc640 10 client.4285.objecter ms_handle_connect 0x7f7fcc06de00 2024-03-25T19:28:47.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.670+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f7fd80548b0 con 0x7f7fe813f520 2024-03-25T19:28:47.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.670+0000 7f7fde7fc640 10 client.4285.objecter ms_dispatch 0x7f7fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:47.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.670+0000 7f7fde7fc640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:47.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.670+0000 7f7fde7fc640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:47.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.671+0000 7f7fde7fc640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-25T19:28:47.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.672+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fd80503c0 con 0x7f7fe813f520 2024-03-25T19:28:47.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.672+0000 7f7fde7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:47.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.672+0000 7f7fde7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:47.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.923+0000 7f7ff0154640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:47.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.923+0000 7f7ff0154640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:47.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.923+0000 7f7ff0154640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:28:47.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.923+0000 7f7ff0154640 1 -- 172.21.15.52:0/2384940924 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fe80627b0 con 0x7f7fe813f520 2024-03-25T19:28:47.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.961+0000 7f7fde7fc640 1 -- 172.21.15.52:0/2384940924 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f7fd80522f0 con 0x7f7fe813f520 2024-03-25T19:28:47.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.961+0000 7f7fde7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:47.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.961+0000 7f7fde7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:47.961 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:28:47.961 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 10 librados: watch_flush enter 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 10 librados: watch_flush exit 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 10 client.4285.objecter successfully canceled tick 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 1 -- 172.21.15.52:0/2384940924 >> v1:172.21.15.52:6800/3967427147 conn(0x7f7fcc06de00 legacy=0x7f7fcc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 10 monclient: shutdown 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 1 -- 172.21.15.52:0/2384940924 >> v1:172.21.15.179:6789/0 conn(0x7f7fe813f520 legacy=0x7f7fe81db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 1 -- 172.21.15.52:0/2384940924 shutdown_connections 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 1 -- 172.21.15.52:0/2384940924 >> 172.21.15.52:0/2384940924 conn(0x7f7fe80b0cc0 msgr2=0x7f7fe80adaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 1 -- 172.21.15.52:0/2384940924 shutdown_connections 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 1 -- 172.21.15.52:0/2384940924 wait complete. 2024-03-25T19:28:47.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:47.963+0000 7f7ff0154640 1 librados: shutdown 2024-03-25T19:28:47.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:54.780 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:54.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.898+0000 7f4e3612b640 10 monclient: get_monmap_and_config 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.898+0000 7f4e3612b640 10 monclient: build_initial_monmap 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.898+0000 7f4e3612b640 10 monclient: monmap: 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:28:54.898981+0000 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:28:54.898981+0000 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:28:54.898 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.899+0000 7f4e3612b640 1 Processor -- start 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.899+0000 7f4e3612b640 1 -- start start 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.899+0000 7f4e3612b640 10 monclient: init 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.899+0000 7f4e3612b640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.899+0000 7f4e3612b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.899+0000 7f4e3612b640 10 monclient(hunting): picked mon.noname-c con 0x7f4e300a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.899+0000 7f4e3612b640 10 monclient(hunting): picked mon.noname-a con 0x7f4e3013f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.900+0000 7f4e3612b640 10 monclient(hunting): picked mon.noname-b con 0x7f4e301430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.900+0000 7f4e3612b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e30142520 con 0x7f4e3013f710 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.900+0000 7f4e3612b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e300b28a0 con 0x7f4e300a98f0 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.900+0000 7f4e3612b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e300b2a30 con 0x7f4e301430c0 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.900+0000 7f4e3612b640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:54.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.900+0000 7f4e3612b640 10 monclient(hunting): authenticate will time out at 5983.079985s 2024-03-25T19:28:54.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.914+0000 7f4e2ffff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f4e301430c0 0x7f4e3014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:52102/0 (socket says 172.21.15.52:52102) 2024-03-25T19:28:54.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.914+0000 7f4e2ffff640 1 -- 172.21.15.52:0/143969232 learned_addr learned my addr 172.21.15.52:0/143969232 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:54.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.922+0000 7f4e2e7fc640 1 -- 172.21.15.52:0/143969232 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578110815 0 0) 0x7f4e300b2a30 con 0x7f4e301430c0 2024-03-25T19:28:54.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.922+0000 7f4e2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:54.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.922+0000 7f4e2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:54.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.922+0000 7f4e2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:54.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.922+0000 7f4e2e7fc640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:28:54.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.922+0000 7f4e2e7fc640 1 -- 172.21.15.52:0/143969232 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e18003520 con 0x7f4e301430c0 2024-03-25T19:28:54.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.925+0000 7f4e2e7fc640 1 -- 172.21.15.52:0/143969232 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840964805 0 0) 0x7f4e300b28a0 con 0x7f4e300a98f0 2024-03-25T19:28:54.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.925+0000 7f4e2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:54.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.925+0000 7f4e2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:54.925 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.925+0000 7f4e2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:54.925 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.925+0000 7f4e2e7fc640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:28:54.925 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.925+0000 7f4e2e7fc640 1 -- 172.21.15.52:0/143969232 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e300b2a30 con 0x7f4e300a98f0 2024-03-25T19:28:54.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 1 -- 172.21.15.52:0/143969232 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1032591561 0 0) 0x7f4e300b2a30 con 0x7f4e300a98f0 2024-03-25T19:28:54.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 1 -- 172.21.15.52:0/143969232 >> v1:172.21.15.52:6790/0 conn(0x7f4e301430c0 legacy=0x7f4e3014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:54.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 1 -- 172.21.15.52:0/143969232 >> v1:172.21.15.52:6789/0 conn(0x7f4e3013f710 legacy=0x7f4e301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:54.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:54.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:28:54.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:54.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:28:54.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 1 -- 172.21.15.52:0/143969232 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e300b44c0 con 0x7f4e300a98f0 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 1 -- 172.21.15.52:0/143969232 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4e20001670 con 0x7f4e300a98f0 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e2e7fc640 10 monclient: dump: 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:54.934 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:54.935 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:54.935 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:54.935 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:54.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e3612b640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:28:54.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.933+0000 7f4e3612b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:54.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.958+0000 7f4e2e7fc640 1 -- 172.21.15.52:0/143969232 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e200017d0 con 0x7f4e300a98f0 2024-03-25T19:28:54.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.958+0000 7f4e2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.958+0000 7f4e2e7fc640 1 -- 172.21.15.52:0/143969232 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4e20001b60 con 0x7f4e300a98f0 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.958+0000 7f4e3612b640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.958+0000 7f4e3612b640 4 set_mon_vals no callback set 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.958+0000 7f4e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.958+0000 7f4e2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.958+0000 7f4e2e7fc640 10 monclient: dump: 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:54.958 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:54.959 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.959+0000 7f4e3612b640 10 monclient: shutdown 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.959+0000 7f4e3612b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.959+0000 7f4e3612b640 1 -- 172.21.15.52:0/143969232 >> v1:172.21.15.179:6789/0 conn(0x7f4e300a98f0 legacy=0x7f4e300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.959+0000 7f4e3612b640 1 -- 172.21.15.52:0/143969232 shutdown_connections 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.959+0000 7f4e3612b640 1 -- 172.21.15.52:0/143969232 >> 172.21.15.52:0/143969232 conn(0x7f4e300b7ea0 msgr2=0x7f4e300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.959+0000 7f4e3612b640 1 -- 172.21.15.52:0/143969232 shutdown_connections 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.959+0000 7f4e3612b640 1 -- 172.21.15.52:0/143969232 wait complete. 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.959+0000 7f4e3612b640 10 monclient: build_initial_monmap 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.959+0000 7f4e3612b640 10 monclient: monmap: 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:28:54.961 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.960+0000 7f4e3612b640 1 librados: starting msgr at 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.960+0000 7f4e3612b640 1 librados: starting objecter 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.960+0000 7f4e3612b640 1 Processor -- start 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.960+0000 7f4e3612b640 1 -- start start 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.960+0000 7f4e3612b640 1 librados: setting wanted keys 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.960+0000 7f4e3612b640 1 librados: calling monclient init 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.960+0000 7f4e3612b640 10 monclient: init 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.961+0000 7f4e3612b640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:54.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.961+0000 7f4e3612b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:54.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.961+0000 7f4e3612b640 10 monclient(hunting): picked mon.noname-c con 0x7f4e300a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:28:54.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.961+0000 7f4e3612b640 10 monclient(hunting): picked mon.noname-b con 0x7f4e3013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:28:54.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.961+0000 7f4e3612b640 10 monclient(hunting): picked mon.noname-a con 0x7f4e301430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:28:54.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.961+0000 7f4e3612b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e301d7870 con 0x7f4e301430c0 2024-03-25T19:28:54.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.961+0000 7f4e3612b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e300abd10 con 0x7f4e3013f710 2024-03-25T19:28:54.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.961+0000 7f4e3612b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e301d7490 con 0x7f4e300a98f0 2024-03-25T19:28:54.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.961+0000 7f4e3612b640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:54.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.961+0000 7f4e3612b640 10 monclient(hunting): authenticate will time out at 5983.141038s 2024-03-25T19:28:54.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.963+0000 7f4e2ffff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f4e301430c0 0x7f4e30134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:52032/0 (socket says 172.21.15.52:52032) 2024-03-25T19:28:54.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.963+0000 7f4e2ffff640 1 -- 172.21.15.52:0/3050115800 learned_addr learned my addr 172.21.15.52:0/3050115800 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:28:54.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.965+0000 7f4e17fff640 10 client.?.objecter ms_handle_connect 0x7f4e301430c0 2024-03-25T19:28:54.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.966+0000 7f4e17fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:54.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.974+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437205124 0 0) 0x7f4e301d7870 con 0x7f4e301430c0 2024-03-25T19:28:54.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.974+0000 7f4e17fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:54.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.974+0000 7f4e17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:54.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.974+0000 7f4e17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:54.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.974+0000 7f4e17fff640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:28:54.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.974+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e1c004eb0 con 0x7f4e301430c0 2024-03-25T19:28:54.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.976+0000 7f4e17fff640 10 client.?.objecter ms_handle_connect 0x7f4e300a98f0 2024-03-25T19:28:54.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.976+0000 7f4e17fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:54.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.980+0000 7f4e17fff640 10 client.?.objecter ms_handle_connect 0x7f4e3013f710 2024-03-25T19:28:54.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.980+0000 7f4e17fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:54.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.990+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072719550 0 0) 0x7f4e301d7490 con 0x7f4e300a98f0 2024-03-25T19:28:54.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.990+0000 7f4e17fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:54.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.990+0000 7f4e17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:54.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.990+0000 7f4e17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:54.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.990+0000 7f4e17fff640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:28:54.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.990+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e301d7870 con 0x7f4e300a98f0 2024-03-25T19:28:54.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.992+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587617537 0 0) 0x7f4e300abd10 con 0x7f4e3013f710 2024-03-25T19:28:54.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.992+0000 7f4e17fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:54.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.992+0000 7f4e17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:54.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.992+0000 7f4e17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:54.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.992+0000 7f4e17fff640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:28:54.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.992+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e301d7490 con 0x7f4e3013f710 2024-03-25T19:28:54.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.993+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1477701423 0 0) 0x7f4e1c004eb0 con 0x7f4e301430c0 2024-03-25T19:28:54.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.993+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e300abd10 con 0x7f4e301430c0 2024-03-25T19:28:54.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.993+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4e28001b30 con 0x7f4e301430c0 2024-03-25T19:28:54.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.993+0000 7f4e17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:54.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.993+0000 7f4e17fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:54.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.993+0000 7f4e17fff640 10 monclient(hunting): dump: 2024-03-25T19:28:54.993 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:54.993 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:54.994 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:54.994 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:54.994 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:54.994 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:54.994 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:54.994 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:54.994 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:54.994 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:54.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.993+0000 7f4e17fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:54.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 623906601 0 0) 0x7f4e300abd10 con 0x7f4e301430c0 2024-03-25T19:28:54.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 >> v1:172.21.15.52:6790/0 conn(0x7f4e300a98f0 legacy=0x7f4e30138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:54.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 >> v1:172.21.15.179:6789/0 conn(0x7f4e3013f710 legacy=0x7f4e301d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:54.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e17fff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:54.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e17fff640 1 monclient: found mon.a 2024-03-25T19:28:54.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:54.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e17fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:54.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e30137a20 con 0x7f4e301430c0 2024-03-25T19:28:54.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e17fff640 10 monclient: _finish_auth 0 2024-03-25T19:28:54.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e17fff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:55.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:55.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e3612b640 5 monclient: authenticate success, global_id 4284 2024-03-25T19:28:55.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e3612b640 10 monclient: _renew_subs 2024-03-25T19:28:55.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e3612b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:55.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e3612b640 1 -- 172.21.15.52:0/3050115800 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e30137c90 con 0x7f4e301430c0 2024-03-25T19:28:55.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e3612b640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:55.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e3612b640 10 monclient: _renew_subs 2024-03-25T19:28:55.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e3612b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:55.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e3612b640 1 -- 172.21.15.52:0/3050115800 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e30134900 con 0x7f4e301430c0 2024-03-25T19:28:55.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:54.999+0000 7f4e3612b640 1 librados: init done 2024-03-25T19:28:55.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.000+0000 7f4e3612b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:55.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.000+0000 7f4e3612b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:55.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.000+0000 7f4e3612b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:55.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.000+0000 7f4e3612b640 1 -- 172.21.15.52:0/3050115800 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e301d7620 con 0x7f4e301430c0 2024-03-25T19:28:55.301 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.301+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e28001fe0 con 0x7f4e301430c0 2024-03-25T19:28:55.301 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.301+0000 7f4e17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.301+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4e28004d00 con 0x7f4e301430c0 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.301+0000 7f4e17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.301+0000 7f4e17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.301+0000 7f4e17fff640 10 monclient: dump: 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:28:55.302 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:28:55.303 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:28:55.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.301+0000 7f4e2dffb640 4 set_mon_vals no callback set 2024-03-25T19:28:55.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.302+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f4e28004f20 con 0x7f4e301430c0 2024-03-25T19:28:55.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.303+0000 7f4e2f7fe640 10 monclient: get_auth_request con 0x7f4e1c06ddf0 auth_method 0 2024-03-25T19:28:55.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.303+0000 7f4e17fff640 10 client.4284.objecter ms_handle_connect 0x7f4e1c06ddf0 2024-03-25T19:28:55.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.310+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f4e28054a70 con 0x7f4e301430c0 2024-03-25T19:28:55.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.310+0000 7f4e17fff640 10 client.4284.objecter ms_dispatch 0x7f4e30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:55.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.310+0000 7f4e17fff640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:55.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.310+0000 7f4e17fff640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:55.309 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.310+0000 7f4e17fff640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-25T19:28:55.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.310+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e28050580 con 0x7f4e301430c0 2024-03-25T19:28:55.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.310+0000 7f4e17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:55.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.310+0000 7f4e17fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:55.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.536+0000 7f4e3612b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:55.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.536+0000 7f4e3612b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:55.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.536+0000 7f4e3612b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:28:55.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.536+0000 7f4e3612b640 1 -- 172.21.15.52:0/3050115800 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e300627b0 con 0x7f4e301430c0 2024-03-25T19:28:55.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.565+0000 7f4e17fff640 1 -- 172.21.15.52:0/3050115800 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f4e280524b0 con 0x7f4e301430c0 2024-03-25T19:28:55.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.565+0000 7f4e17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:55.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.565+0000 7f4e17fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:55.565 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:28:55.565 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:55.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 10 librados: watch_flush enter 2024-03-25T19:28:55.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 10 librados: watch_flush exit 2024-03-25T19:28:55.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-25T19:28:55.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 10 client.4284.objecter successfully canceled tick 2024-03-25T19:28:55.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 1 -- 172.21.15.52:0/3050115800 >> v1:172.21.15.52:6800/3967427147 conn(0x7f4e1c06ddf0 legacy=0x7f4e1c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 10 monclient: shutdown 2024-03-25T19:28:55.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:55.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 1 -- 172.21.15.52:0/3050115800 >> v1:172.21.15.52:6789/0 conn(0x7f4e301430c0 legacy=0x7f4e30134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:55.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 1 -- 172.21.15.52:0/3050115800 shutdown_connections 2024-03-25T19:28:55.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 1 -- 172.21.15.52:0/3050115800 >> 172.21.15.52:0/3050115800 conn(0x7f4e300b7ea0 msgr2=0x7f4e30074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:55.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 1 -- 172.21.15.52:0/3050115800 shutdown_connections 2024-03-25T19:28:55.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.567+0000 7f4e3612b640 1 -- 172.21.15.52:0/3050115800 wait complete. 2024-03-25T19:28:55.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:28:55.568+0000 7f4e3612b640 1 librados: shutdown 2024-03-25T19:28:55.579 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:01.106 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:01:54.705 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:01.169 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:01:54.768 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:01.193 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:01:54.792 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:01.207 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:01:54.806 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:01.218 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:01:54.817 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:01.224 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:01:54.764 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:01.227 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:01:54.769 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:01.242 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:01:54.841 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:01.256 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:01:54.797 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:02.387 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:02.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.500+0000 7f53fb52b640 10 monclient: get_monmap_and_config 2024-03-25T19:29:02.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.500+0000 7f53fb52b640 10 monclient: build_initial_monmap 2024-03-25T19:29:02.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.501+0000 7f53fb52b640 10 monclient: monmap: 2024-03-25T19:29:02.501 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:02.501 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:02.501 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:29:02.501578+0000 2024-03-25T19:29:02.501 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:29:02.501578+0000 2024-03-25T19:29:02.501 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:02.501 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:02.501 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:02.501 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:29:02.501 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:29:02.501 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:02.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.501+0000 7f53fb52b640 1 Processor -- start 2024-03-25T19:29:02.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.501+0000 7f53fb52b640 1 -- start start 2024-03-25T19:29:02.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.501+0000 7f53fb52b640 10 monclient: init 2024-03-25T19:29:02.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.502+0000 7f53fb52b640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:02.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.502+0000 7f53fb52b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:02.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.502+0000 7f53fb52b640 10 monclient(hunting): picked mon.noname-a con 0x7f53f413fc60 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:02.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.502+0000 7f53fb52b640 10 monclient(hunting): picked mon.noname-b con 0x7f53f4070570 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:02.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.502+0000 7f53fb52b640 10 monclient(hunting): picked mon.noname-c con 0x7f53f40727a0 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:02.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.502+0000 7f53fb52b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f413c9f0 con 0x7f53f413fc60 2024-03-25T19:29:02.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.503+0000 7f53fb52b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f406fac0 con 0x7f53f40727a0 2024-03-25T19:29:02.503 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.503+0000 7f53fb52b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f40715d0 con 0x7f53f4070570 2024-03-25T19:29:02.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.503+0000 7f53fb52b640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:02.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.503+0000 7f53fb52b640 10 monclient(hunting): authenticate will time out at 5990.683029s 2024-03-25T19:29:02.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.511+0000 7f53f8ac6640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f53f4070570 0x7f53f4072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:56668/0 (socket says 172.21.15.52:56668) 2024-03-25T19:29:02.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.511+0000 7f53f8ac6640 1 -- 172.21.15.52:0/2820987099 learned_addr learned my addr 172.21.15.52:0/2820987099 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:02.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.524+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242839372 0 0) 0x7f53f40715d0 con 0x7f53f4070570 2024-03-25T19:29:02.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.524+0000 7f53ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:02.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.524+0000 7f53ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:02.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.524+0000 7f53ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:02.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.524+0000 7f53ebfff640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:29:02.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.524+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e0003520 con 0x7f53f4070570 2024-03-25T19:29:02.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.533+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939054065 0 0) 0x7f53f413c9f0 con 0x7f53f413fc60 2024-03-25T19:29:02.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.533+0000 7f53ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:02.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.533+0000 7f53ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:02.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.533+0000 7f53ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:02.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.533+0000 7f53ebfff640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:29:02.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.533+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f40715d0 con 0x7f53f413fc60 2024-03-25T19:29:02.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.541+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914142542 0 0) 0x7f53f406fac0 con 0x7f53f40727a0 2024-03-25T19:29:02.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.541+0000 7f53ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:02.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.541+0000 7f53ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:02.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.541+0000 7f53ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:02.540 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.541+0000 7f53ebfff640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:29:02.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.541+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f413c9f0 con 0x7f53f40727a0 2024-03-25T19:29:02.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3264356435 0 0) 0x7f53f40715d0 con 0x7f53f413fc60 2024-03-25T19:29:02.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 >> v1:172.21.15.52:6790/0 conn(0x7f53f4070570 legacy=0x7f53f4072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 >> v1:172.21.15.179:6789/0 conn(0x7f53f40727a0 legacy=0x7f53f414a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:02.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 1 monclient: found mon.noname-a 2024-03-25T19:29:02.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:02.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f40718b0 con 0x7f53f413fc60 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 10 monclient: _finish_auth 0 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f53f0001670 con 0x7f53f413fc60 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53ebfff640 10 monclient: dump: 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.543 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:02.544 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:02.544 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:02.544 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:02.544 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:02.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53fb52b640 5 monclient: authenticate success, global_id 4287 2024-03-25T19:29:02.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.542+0000 7f53fb52b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:02.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53f0001850 con 0x7f53f413fc60 2024-03-25T19:29:02.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:02.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53ebfff640 1 -- 172.21.15.52:0/2820987099 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f53f0001c60 con 0x7f53f413fc60 2024-03-25T19:29:02.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53fb52b640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:02.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53fb52b640 4 set_mon_vals no callback set 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53ebfff640 10 monclient: dump: 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:02.556 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:02.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53fb52b640 10 monclient: shutdown 2024-03-25T19:29:02.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53fb52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:02.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53fb52b640 1 -- 172.21.15.52:0/2820987099 >> v1:172.21.15.52:6789/0 conn(0x7f53f413fc60 legacy=0x7f53f406f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53fb52b640 1 -- 172.21.15.52:0/2820987099 shutdown_connections 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53fb52b640 1 -- 172.21.15.52:0/2820987099 >> 172.21.15.52:0/2820987099 conn(0x7f53f40b4a80 msgr2=0x7f53f40b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53fb52b640 1 -- 172.21.15.52:0/2820987099 shutdown_connections 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.556+0000 7f53fb52b640 1 -- 172.21.15.52:0/2820987099 wait complete. 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.557+0000 7f53fb52b640 10 monclient: build_initial_monmap 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.557+0000 7f53fb52b640 10 monclient: monmap: 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:29:02.559 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.557+0000 7f53fb52b640 1 librados: starting msgr at 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.557+0000 7f53fb52b640 1 librados: starting objecter 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.557+0000 7f53fb52b640 1 Processor -- start 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 1 -- start start 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 1 librados: setting wanted keys 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 1 librados: calling monclient init 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 10 monclient: init 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 10 monclient(hunting): picked mon.noname-b con 0x7f53f4070570 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 10 monclient(hunting): picked mon.noname-c con 0x7f53f40727a0 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 10 monclient(hunting): picked mon.noname-a con 0x7f53f413fc60 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f41d3090 con 0x7f53f413fc60 2024-03-25T19:29:02.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f41421f0 con 0x7f53f4070570 2024-03-25T19:29:02.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f4142380 con 0x7f53f40727a0 2024-03-25T19:29:02.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:02.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.558+0000 7f53fb52b640 10 monclient(hunting): authenticate will time out at 5990.738675s 2024-03-25T19:29:02.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.571+0000 7f53f9ac8640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f53f413fc60 0x7f53f4134d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:56456/0 (socket says 172.21.15.52:56456) 2024-03-25T19:29:02.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.571+0000 7f53f9ac8640 1 -- 172.21.15.52:0/4183352740 learned_addr learned my addr 172.21.15.52:0/4183352740 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:02.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.572+0000 7f53e9ffb640 10 client.?.objecter ms_handle_connect 0x7f53f413fc60 2024-03-25T19:29:02.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.572+0000 7f53e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:02.573 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.574+0000 7f53e9ffb640 10 client.?.objecter ms_handle_connect 0x7f53f4070570 2024-03-25T19:29:02.573 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.574+0000 7f53e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:02.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.575+0000 7f53e9ffb640 10 client.?.objecter ms_handle_connect 0x7f53f40727a0 2024-03-25T19:29:02.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.575+0000 7f53e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:02.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.584+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132197207 0 0) 0x7f53f41d3090 con 0x7f53f413fc60 2024-03-25T19:29:02.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.584+0000 7f53e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:02.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.584+0000 7f53e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:02.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.584+0000 7f53e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:02.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.584+0000 7f53e9ffb640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:29:02.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.584+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d8004eb0 con 0x7f53f413fc60 2024-03-25T19:29:02.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.593+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846772707 0 0) 0x7f53f4142380 con 0x7f53f40727a0 2024-03-25T19:29:02.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.593+0000 7f53e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:02.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.593+0000 7f53e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:02.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.593+0000 7f53e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:02.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.593+0000 7f53e9ffb640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:29:02.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.593+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f41d3090 con 0x7f53f40727a0 2024-03-25T19:29:02.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.594+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11222298 0 0) 0x7f53f41421f0 con 0x7f53f4070570 2024-03-25T19:29:02.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.594+0000 7f53e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:02.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.594+0000 7f53e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:02.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.594+0000 7f53e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:02.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.594+0000 7f53e9ffb640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:29:02.594 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.594+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f4142380 con 0x7f53f4070570 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.612+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228487368 0 0) 0x7f53f4142380 con 0x7f53f4070570 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.612+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53f41421f0 con 0x7f53f4070570 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.612+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f53f0001b60 con 0x7f53f4070570 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.612+0000 7f53e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.612+0000 7f53e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.612+0000 7f53e9ffb640 10 monclient(hunting): dump: 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:02.612 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:02.613 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:02.613 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:02.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.612+0000 7f53e9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.615+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3660770659 0 0) 0x7f53d8004eb0 con 0x7f53f413fc60 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.615+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53f4142380 con 0x7f53f413fc60 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.615+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f53ec001cd0 con 0x7f53f413fc60 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.615+0000 7f53e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.615+0000 7f53e9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.615+0000 7f53e9ffb640 10 monclient(hunting): dump: 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:02.615 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:02.616 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:02.616 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:02.616 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.626+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1974397922 0 0) 0x7f53f41d3090 con 0x7f53f40727a0 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.626+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53d8004eb0 con 0x7f53f40727a0 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.626+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f53e4001950 con 0x7f53f40727a0 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.626+0000 7f53e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.626+0000 7f53e9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.626+0000 7f53e9ffb640 10 monclient(hunting): dump: 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.626 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:02.627 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:02.627 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:02.627 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:02.627 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4287498106 0 0) 0x7f53f4142380 con 0x7f53f413fc60 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 >> v1:172.21.15.52:6790/0 conn(0x7f53f40727a0 legacy=0x7f53f41d35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 >> v1:172.21.15.179:6789/0 conn(0x7f53f4070570 legacy=0x7f53f4141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53e9ffb640 1 monclient: found mon.a 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53e9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f4137070 con 0x7f53f413fc60 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53e9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53fb52b640 5 monclient: authenticate success, global_id 4290 2024-03-25T19:29:02.631 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53fb52b640 10 monclient: _renew_subs 2024-03-25T19:29:02.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53fb52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:29:02.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53fb52b640 1 -- 172.21.15.52:0/4183352740 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f41d3cc0 con 0x7f53f413fc60 2024-03-25T19:29:02.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53fb52b640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:02.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53fb52b640 10 monclient: _renew_subs 2024-03-25T19:29:02.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53fb52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:29:02.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53fb52b640 1 -- 172.21.15.52:0/4183352740 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f41414b0 con 0x7f53f413fc60 2024-03-25T19:29:02.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.631+0000 7f53fb52b640 1 librados: init done 2024-03-25T19:29:02.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.633+0000 7f53fb52b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:02.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.633+0000 7f53fb52b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:02.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.633+0000 7f53fb52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:29:02.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.633+0000 7f53fb52b640 1 -- 172.21.15.52:0/4183352740 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f4137260 con 0x7f53f413fc60 2024-03-25T19:29:02.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.940+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53ec003990 con 0x7f53f413fc60 2024-03-25T19:29:02.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.940+0000 7f53e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:02.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.940+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f53ec003da0 con 0x7f53f413fc60 2024-03-25T19:29:02.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.940+0000 7f53e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:02.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.940+0000 7f53e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:02.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.940+0000 7f53e9ffb640 10 monclient: dump: 2024-03-25T19:29:02.940 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:02.941 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:02.941 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.941 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:02.941 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:02.941 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:02.941 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:02.941 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:02.941 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:02.941 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:02.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.940+0000 7f53eb7fe640 4 set_mon_vals no callback set 2024-03-25T19:29:02.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.941+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f53ec003fc0 con 0x7f53f413fc60 2024-03-25T19:29:02.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.942+0000 7f53f92c7640 10 monclient: get_auth_request con 0x7f53d806e160 auth_method 0 2024-03-25T19:29:02.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.942+0000 7f53e9ffb640 10 client.4290.objecter ms_handle_connect 0x7f53d806e160 2024-03-25T19:29:02.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.947+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f53ec054b30 con 0x7f53f413fc60 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.947+0000 7f53e9ffb640 10 client.4290.objecter ms_dispatch 0x7f53f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.947+0000 7f53e9ffb640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.947+0000 7f53e9ffb640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.947+0000 7f53e9ffb640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.947+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53ec050640 con 0x7f53f413fc60 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.947+0000 7f53e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:02.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:02.947+0000 7f53e9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:03.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.186+0000 7f53fb52b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:03.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.186+0000 7f53fb52b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:03.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.186+0000 7f53fb52b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:29:03.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.186+0000 7f53fb52b640 1 -- 172.21.15.52:0/4183352740 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53f41d3220 con 0x7f53f413fc60 2024-03-25T19:29:03.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.222+0000 7f53e9ffb640 1 -- 172.21.15.52:0/4183352740 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f53ec052570 con 0x7f53f413fc60 2024-03-25T19:29:03.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.222+0000 7f53e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:03.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.222+0000 7f53e9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:03.221 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:29:03.222 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:03.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 10 librados: watch_flush enter 2024-03-25T19:29:03.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 10 librados: watch_flush exit 2024-03-25T19:29:03.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-25T19:29:03.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 10 client.4290.objecter successfully canceled tick 2024-03-25T19:29:03.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 1 -- 172.21.15.52:0/4183352740 >> v1:172.21.15.52:6800/3967427147 conn(0x7f53d806e160 legacy=0x7f53d8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:03.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 10 monclient: shutdown 2024-03-25T19:29:03.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:03.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 1 -- 172.21.15.52:0/4183352740 >> v1:172.21.15.52:6789/0 conn(0x7f53f413fc60 legacy=0x7f53f4134d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:03.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 1 -- 172.21.15.52:0/4183352740 shutdown_connections 2024-03-25T19:29:03.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 1 -- 172.21.15.52:0/4183352740 >> 172.21.15.52:0/4183352740 conn(0x7f53f40b4a80 msgr2=0x7f53f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:03.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 1 -- 172.21.15.52:0/4183352740 shutdown_connections 2024-03-25T19:29:03.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 1 -- 172.21.15.52:0/4183352740 wait complete. 2024-03-25T19:29:03.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:03.224+0000 7f53fb52b640 1 librados: shutdown 2024-03-25T19:29:03.234 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:10.042 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:10.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.156+0000 7f31e20ba640 10 monclient: get_monmap_and_config 2024-03-25T19:29:10.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.156+0000 7f31e20ba640 10 monclient: build_initial_monmap 2024-03-25T19:29:10.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.156+0000 7f31e20ba640 10 monclient: monmap: 2024-03-25T19:29:10.156 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:10.156 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:10.156 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:29:10.157366+0000 2024-03-25T19:29:10.156 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:29:10.157366+0000 2024-03-25T19:29:10.156 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:10.157 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:10.157 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:10.157 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:29:10.157 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:29:10.157 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:10.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.157+0000 7f31e20ba640 1 Processor -- start 2024-03-25T19:29:10.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.157+0000 7f31e20ba640 1 -- start start 2024-03-25T19:29:10.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.157+0000 7f31e20ba640 10 monclient: init 2024-03-25T19:29:10.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.158+0000 7f31e20ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:10.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.158+0000 7f31e20ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:10.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.158+0000 7f31e20ba640 10 monclient(hunting): picked mon.noname-a con 0x7f31dc0b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:10.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.158+0000 7f31e20ba640 10 monclient(hunting): picked mon.noname-c con 0x7f31dc0b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:10.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.158+0000 7f31e20ba640 10 monclient(hunting): picked mon.noname-b con 0x7f31dc14be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:10.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.158+0000 7f31e20ba640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc0af940 con 0x7f31dc0b4d60 2024-03-25T19:29:10.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.158+0000 7f31e20ba640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc0b0a70 con 0x7f31dc0b1730 2024-03-25T19:29:10.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.158+0000 7f31e20ba640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc0b0d50 con 0x7f31dc14be90 2024-03-25T19:29:10.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.158+0000 7f31e20ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:10.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.158+0000 7f31e20ba640 10 monclient(hunting): authenticate will time out at 5998.338578s 2024-03-25T19:29:10.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.170+0000 7f31db7fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f31dc0b4d60 0x7f31dc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:57448/0 (socket says 172.21.15.52:57448) 2024-03-25T19:29:10.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.170+0000 7f31db7fe640 1 -- 172.21.15.52:0/3285566769 learned_addr learned my addr 172.21.15.52:0/3285566769 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:10.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.179+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784806264 0 0) 0x7f31dc0af940 con 0x7f31dc0b4d60 2024-03-25T19:29:10.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.179+0000 7f31da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:10.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.179+0000 7f31da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:10.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.179+0000 7f31da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:10.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.179+0000 7f31da7fc640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:29:10.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.179+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c4003520 con 0x7f31dc0b4d60 2024-03-25T19:29:10.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.187+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201177310 0 0) 0x7f31dc0b0a70 con 0x7f31dc0b1730 2024-03-25T19:29:10.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.187+0000 7f31da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:10.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.187+0000 7f31da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:10.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.187+0000 7f31da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:10.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.187+0000 7f31da7fc640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:29:10.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.187+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31dc0af940 con 0x7f31dc0b1730 2024-03-25T19:29:10.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.188+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012928846 0 0) 0x7f31dc0b0d50 con 0x7f31dc14be90 2024-03-25T19:29:10.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.188+0000 7f31da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.188+0000 7f31da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.188+0000 7f31da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.188+0000 7f31da7fc640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.188+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31dc0b0a70 con 0x7f31dc14be90 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494684358 0 0) 0x7f31dc0af940 con 0x7f31dc0b1730 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 >> v1:172.21.15.52:6790/0 conn(0x7f31dc14be90 legacy=0x7f31dc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 >> v1:172.21.15.52:6789/0 conn(0x7f31dc0b4d60 legacy=0x7f31dc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 1 monclient: found mon.noname-c 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc0b1030 con 0x7f31dc0b1730 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 10 monclient: _finish_auth 0 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:29:10.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f31d0001670 con 0x7f31dc0b1730 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31da7fc640 10 monclient: dump: 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:10.221 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31e20ba640 5 monclient: authenticate success, global_id 4300 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.198+0000 7f31e20ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.206+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31d00017d0 con 0x7f31dc0b1730 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.206+0000 7f31da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.206+0000 7f31da7fc640 1 -- 172.21.15.52:0/3285566769 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f31d0001be0 con 0x7f31dc0b1730 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.206+0000 7f31da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.206+0000 7f31da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.206+0000 7f31da7fc640 10 monclient: dump: 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:10.222 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.206+0000 7f31e20ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.206+0000 7f31e20ba640 4 set_mon_vals no callback set 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.207+0000 7f31e20ba640 10 monclient: shutdown 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.207+0000 7f31e20ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.207+0000 7f31e20ba640 1 -- 172.21.15.52:0/3285566769 >> v1:172.21.15.179:6789/0 conn(0x7f31dc0b1730 legacy=0x7f31dc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.207+0000 7f31e20ba640 1 -- 172.21.15.52:0/3285566769 shutdown_connections 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.207+0000 7f31e20ba640 1 -- 172.21.15.52:0/3285566769 >> 172.21.15.52:0/3285566769 conn(0x7f31dc0617a0 msgr2=0x7f31dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.207+0000 7f31e20ba640 1 -- 172.21.15.52:0/3285566769 shutdown_connections 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.207+0000 7f31e20ba640 1 -- 172.21.15.52:0/3285566769 wait complete. 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.208+0000 7f31e20ba640 10 monclient: build_initial_monmap 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.208+0000 7f31e20ba640 10 monclient: monmap: 2024-03-25T19:29:10.223 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.208+0000 7f31e20ba640 1 librados: starting msgr at 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.208+0000 7f31e20ba640 1 librados: starting objecter 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.209+0000 7f31e20ba640 1 Processor -- start 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.209+0000 7f31e20ba640 1 -- start start 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.209+0000 7f31e20ba640 1 librados: setting wanted keys 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.209+0000 7f31e20ba640 1 librados: calling monclient init 2024-03-25T19:29:10.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.209+0000 7f31e20ba640 10 monclient: init 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.210+0000 7f31e20ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.210+0000 7f31e20ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.210+0000 7f31e20ba640 10 monclient(hunting): picked mon.noname-b con 0x7f31dc0b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.210+0000 7f31e20ba640 10 monclient(hunting): picked mon.noname-a con 0x7f31dc0b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.210+0000 7f31e20ba640 10 monclient(hunting): picked mon.noname-c con 0x7f31dc14be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.210+0000 7f31e20ba640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc1db0a0 con 0x7f31dc0b4d60 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.210+0000 7f31e20ba640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc1dace0 con 0x7f31dc0b1730 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.210+0000 7f31e20ba640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc1dae70 con 0x7f31dc14be90 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.210+0000 7f31e20ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.210+0000 7f31e20ba640 10 monclient(hunting): authenticate will time out at 5998.390143s 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.219+0000 7f31daffd640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f31dc0b4d60 0x7f31dc1db5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:57492/0 (socket says 172.21.15.52:57492) 2024-03-25T19:29:10.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.219+0000 7f31daffd640 1 -- 172.21.15.52:0/3901383683 learned_addr learned my addr 172.21.15.52:0/3901383683 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:10.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.220+0000 7f31c3fff640 10 client.?.objecter ms_handle_connect 0x7f31dc0b4d60 2024-03-25T19:29:10.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.220+0000 7f31c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:10.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.234+0000 7f31c3fff640 10 client.?.objecter ms_handle_connect 0x7f31dc0b1730 2024-03-25T19:29:10.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.234+0000 7f31c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:10.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.235+0000 7f31c3fff640 10 client.?.objecter ms_handle_connect 0x7f31dc14be90 2024-03-25T19:29:10.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.235+0000 7f31c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:10.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.242+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953517626 0 0) 0x7f31dc1dace0 con 0x7f31dc0b1730 2024-03-25T19:29:10.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.242+0000 7f31c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:10.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.242+0000 7f31c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:10.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.242+0000 7f31c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:10.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.242+0000 7f31c3fff640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:29:10.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.242+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31c8004eb0 con 0x7f31dc0b1730 2024-03-25T19:29:10.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.249+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662910532 0 0) 0x7f31dc1dae70 con 0x7f31dc14be90 2024-03-25T19:29:10.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.249+0000 7f31c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:10.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.249+0000 7f31c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:10.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.249+0000 7f31c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:10.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.249+0000 7f31c3fff640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:29:10.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.249+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31dc1dace0 con 0x7f31dc14be90 2024-03-25T19:29:10.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.255+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 750447839 0 0) 0x7f31c8004eb0 con 0x7f31dc0b1730 2024-03-25T19:29:10.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.255+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31dc1dae70 con 0x7f31dc0b1730 2024-03-25T19:29:10.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.255+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f31cc001950 con 0x7f31dc0b1730 2024-03-25T19:29:10.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.255+0000 7f31c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:10.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.255+0000 7f31c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:10.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.255+0000 7f31c3fff640 10 monclient(hunting): dump: 2024-03-25T19:29:10.255 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:10.255 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:10.255 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:10.256 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:10.256 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:10.256 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:10.256 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:10.256 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:10.256 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:10.256 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:10.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.255+0000 7f31c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:10.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3550801719 0 0) 0x7f31dc1dae70 con 0x7f31dc0b1730 2024-03-25T19:29:10.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 >> v1:172.21.15.52:6790/0 conn(0x7f31dc14be90 legacy=0x7f31dc13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 >> v1:172.21.15.52:6789/0 conn(0x7f31dc0b4d60 legacy=0x7f31dc1db5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31c3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:10.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31c3fff640 1 monclient: found mon.b 2024-03-25T19:29:10.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:10.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:10.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc13b9d0 con 0x7f31dc0b1730 2024-03-25T19:29:10.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31c3fff640 10 monclient: _finish_auth 0 2024-03-25T19:29:10.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31c3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:10.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:10.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31e20ba640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:29:10.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31e20ba640 10 monclient: _renew_subs 2024-03-25T19:29:10.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31e20ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:10.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31e20ba640 1 -- 172.21.15.52:0/3901383683 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31dc13bc20 con 0x7f31dc0b1730 2024-03-25T19:29:10.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31e20ba640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:10.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31e20ba640 10 monclient: _renew_subs 2024-03-25T19:29:10.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31e20ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31e20ba640 1 -- 172.21.15.52:0/3901383683 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc1429b0 con 0x7f31dc0b1730 2024-03-25T19:29:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.263+0000 7f31e20ba640 1 librados: init done 2024-03-25T19:29:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.264+0000 7f31e20ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.264+0000 7f31e20ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.264+0000 7f31e20ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:10.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.264+0000 7f31e20ba640 1 -- 172.21.15.52:0/3901383683 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31dc1db230 con 0x7f31dc0b1730 2024-03-25T19:29:10.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.530+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31cc001d60 con 0x7f31dc0b1730 2024-03-25T19:29:10.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.530+0000 7f31c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:10.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.530+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f31cc004980 con 0x7f31dc0b1730 2024-03-25T19:29:10.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.530+0000 7f31c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:10.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.530+0000 7f31c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:10.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.530+0000 7f31c3fff640 10 monclient: dump: 2024-03-25T19:29:10.531 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.530+0000 7f31d9ffb640 4 set_mon_vals no callback set 2024-03-25T19:29:10.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.531+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f31cc004ba0 con 0x7f31dc0b1730 2024-03-25T19:29:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.532+0000 7f31daffd640 10 monclient: get_auth_request con 0x7f31c806dd50 auth_method 0 2024-03-25T19:29:10.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.532+0000 7f31c3fff640 10 client.4303.objecter ms_handle_connect 0x7f31c806dd50 2024-03-25T19:29:10.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.537+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f31cc0547d0 con 0x7f31dc0b1730 2024-03-25T19:29:10.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.537+0000 7f31c3fff640 10 client.4303.objecter ms_dispatch 0x7f31dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:10.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.537+0000 7f31c3fff640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:10.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.537+0000 7f31c3fff640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:10.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.537+0000 7f31c3fff640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-25T19:29:10.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.538+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31cc050110 con 0x7f31dc0b1730 2024-03-25T19:29:10.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.538+0000 7f31c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:10.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.538+0000 7f31c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:10.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.803+0000 7f31e20ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:10.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.803+0000 7f31e20ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:10.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.803+0000 7f31e20ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:10.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.803+0000 7f31e20ba640 1 -- 172.21.15.52:0/3901383683 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31dc0627b0 con 0x7f31dc0b1730 2024-03-25T19:29:10.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.947+0000 7f31c3fff640 1 -- 172.21.15.52:0/3901383683 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f31cc052210 con 0x7f31dc0b1730 2024-03-25T19:29:10.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.947+0000 7f31c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:10.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.947+0000 7f31c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:10.947 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:29:10.947 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:10.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.949+0000 7f31e20ba640 10 librados: watch_flush enter 2024-03-25T19:29:10.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 10 librados: watch_flush exit 2024-03-25T19:29:10.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-25T19:29:10.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 10 client.4303.objecter successfully canceled tick 2024-03-25T19:29:10.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 1 -- 172.21.15.52:0/3901383683 >> v1:172.21.15.52:6800/3967427147 conn(0x7f31c806dd50 legacy=0x7f31c8070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 10 monclient: shutdown 2024-03-25T19:29:10.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:10.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 1 -- 172.21.15.52:0/3901383683 >> v1:172.21.15.179:6789/0 conn(0x7f31dc0b1730 legacy=0x7f31dc13a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:10.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 1 -- 172.21.15.52:0/3901383683 shutdown_connections 2024-03-25T19:29:10.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 1 -- 172.21.15.52:0/3901383683 >> 172.21.15.52:0/3901383683 conn(0x7f31dc0617a0 msgr2=0x7f31dc13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:10.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 1 -- 172.21.15.52:0/3901383683 shutdown_connections 2024-03-25T19:29:10.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 1 -- 172.21.15.52:0/3901383683 wait complete. 2024-03-25T19:29:10.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:10.950+0000 7f31e20ba640 1 librados: shutdown 2024-03-25T19:29:10.960 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:17.767 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:17.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.885+0000 7f90b7089640 10 monclient: get_monmap_and_config 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.885+0000 7f90b7089640 10 monclient: build_initial_monmap 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.885+0000 7f90b7089640 10 monclient: monmap: 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:29:17.885956+0000 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:29:17.885956+0000 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:29:17.885 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:17.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.886+0000 7f90b7089640 1 Processor -- start 2024-03-25T19:29:17.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.886+0000 7f90b7089640 1 -- start start 2024-03-25T19:29:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.886+0000 7f90b7089640 10 monclient: init 2024-03-25T19:29:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.886+0000 7f90b7089640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.886+0000 7f90b7089640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.886+0000 7f90b7089640 10 monclient(hunting): picked mon.noname-b con 0x7f90b00b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.887+0000 7f90b7089640 10 monclient(hunting): picked mon.noname-c con 0x7f90b00b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.887+0000 7f90b7089640 10 monclient(hunting): picked mon.noname-a con 0x7f90b014be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.887+0000 7f90b7089640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b00af940 con 0x7f90b014be90 2024-03-25T19:29:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.887+0000 7f90b7089640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b00b0a70 con 0x7f90b00b1730 2024-03-25T19:29:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.887+0000 7f90b7089640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b00b0d50 con 0x7f90b00b4d60 2024-03-25T19:29:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.887+0000 7f90b7089640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:17.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.887+0000 7f90b7089640 10 monclient(hunting): authenticate will time out at 6006.066991s 2024-03-25T19:29:17.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.890+0000 7f90affff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f90b00b1730 0x7f90b00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:56702/0 (socket says 172.21.15.52:56702) 2024-03-25T19:29:17.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.890+0000 7f90affff640 1 -- 172.21.15.52:0/3490553457 learned_addr learned my addr 172.21.15.52:0/3490553457 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:17.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.899+0000 7f90af7fe640 1 -- 172.21.15.52:0/3490553457 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871828519 0 0) 0x7f90b00b0a70 con 0x7f90b00b1730 2024-03-25T19:29:17.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.899+0000 7f90af7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:17.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.899+0000 7f90af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:17.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.899+0000 7f90af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:17.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.899+0000 7f90af7fe640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:29:17.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.899+0000 7f90af7fe640 1 -- 172.21.15.52:0/3490553457 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a0003520 con 0x7f90b00b1730 2024-03-25T19:29:17.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 1 -- 172.21.15.52:0/3490553457 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1338693265 0 0) 0x7f90a0003520 con 0x7f90b00b1730 2024-03-25T19:29:17.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 1 -- 172.21.15.52:0/3490553457 >> v1:172.21.15.52:6790/0 conn(0x7f90b00b4d60 legacy=0x7f90b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:17.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 1 -- 172.21.15.52:0/3490553457 >> v1:172.21.15.52:6789/0 conn(0x7f90b014be90 legacy=0x7f90b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:29:17.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 1 monclient: found mon.noname-c 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 1 -- 172.21.15.52:0/3490553457 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b00b1030 con 0x7f90b00b1730 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 10 monclient: _finish_auth 0 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 1 -- 172.21.15.52:0/3490553457 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f909c001690 con 0x7f90b00b1730 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.911+0000 7f90af7fe640 10 monclient: dump: 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:17.912 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:17.913 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:17.913 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:17.913 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:17.913 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:17.913 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:17.913 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:17.913 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:17.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.912+0000 7f90b7089640 5 monclient: authenticate success, global_id 4306 2024-03-25T19:29:17.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.912+0000 7f90b7089640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:17.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.928+0000 7f90af7fe640 1 -- 172.21.15.52:0/3490553457 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f909c001860 con 0x7f90b00b1730 2024-03-25T19:29:17.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.928+0000 7f90af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:17.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.928+0000 7f90af7fe640 1 -- 172.21.15.52:0/3490553457 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f909c001cb0 con 0x7f90b00b1730 2024-03-25T19:29:17.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.928+0000 7f90b7089640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:17.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.928+0000 7f90b7089640 4 set_mon_vals no callback set 2024-03-25T19:29:17.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.928+0000 7f90af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:17.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.928+0000 7f90af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:17.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.928+0000 7f90af7fe640 10 monclient: dump: 2024-03-25T19:29:17.928 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:17.928 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:17.929 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:17.929 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:17.929 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:17.929 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:17.929 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:17.929 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:17.929 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:17.929 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:17.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.929+0000 7f90b7089640 10 monclient: shutdown 2024-03-25T19:29:17.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.929+0000 7f90b7089640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:17.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.929+0000 7f90b7089640 1 -- 172.21.15.52:0/3490553457 >> v1:172.21.15.179:6789/0 conn(0x7f90b00b1730 legacy=0x7f90b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:17.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.929+0000 7f90b7089640 1 -- 172.21.15.52:0/3490553457 shutdown_connections 2024-03-25T19:29:17.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.929+0000 7f90b7089640 1 -- 172.21.15.52:0/3490553457 >> 172.21.15.52:0/3490553457 conn(0x7f90b00617a0 msgr2=0x7f90b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:17.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.929+0000 7f90b7089640 1 -- 172.21.15.52:0/3490553457 shutdown_connections 2024-03-25T19:29:17.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.929+0000 7f90b7089640 1 -- 172.21.15.52:0/3490553457 wait complete. 2024-03-25T19:29:17.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.930+0000 7f90b7089640 10 monclient: build_initial_monmap 2024-03-25T19:29:17.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.930+0000 7f90b7089640 10 monclient: monmap: 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.930+0000 7f90b7089640 1 librados: starting msgr at 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.930+0000 7f90b7089640 1 librados: starting objecter 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.930+0000 7f90b7089640 1 Processor -- start 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 1 -- start start 2024-03-25T19:29:17.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 1 librados: setting wanted keys 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 1 librados: calling monclient init 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 10 monclient: init 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 10 monclient(hunting): picked mon.noname-a con 0x7f90b00b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 10 monclient(hunting): picked mon.noname-c con 0x7f90b00b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 10 monclient(hunting): picked mon.noname-b con 0x7f90b014be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b01dadd0 con 0x7f90b00b1730 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b01381a0 con 0x7f90b014be90 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b0143900 con 0x7f90b00b4d60 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:17.933 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.931+0000 7f90b7089640 10 monclient(hunting): authenticate will time out at 6006.111852s 2024-03-25T19:29:17.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.939+0000 7f90affff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f90b00b4d60 0x7f90b01d7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:54032/0 (socket says 172.21.15.52:54032) 2024-03-25T19:29:17.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.939+0000 7f90affff640 1 -- 172.21.15.52:0/2681468544 learned_addr learned my addr 172.21.15.52:0/2681468544 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:17.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.948+0000 7f90ad7fa640 10 client.?.objecter ms_handle_connect 0x7f90b00b4d60 2024-03-25T19:29:17.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.948+0000 7f90ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:17.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.949+0000 7f90ad7fa640 10 client.?.objecter ms_handle_connect 0x7f90b014be90 2024-03-25T19:29:17.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.949+0000 7f90ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:17.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.953+0000 7f90ad7fa640 10 client.?.objecter ms_handle_connect 0x7f90b00b1730 2024-03-25T19:29:17.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.953+0000 7f90ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:17.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.956+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769917977 0 0) 0x7f90b0143900 con 0x7f90b00b4d60 2024-03-25T19:29:17.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.956+0000 7f90ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:17.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.956+0000 7f90ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:17.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.956+0000 7f90ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:17.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.956+0000 7f90ad7fa640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:29:17.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.956+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9094004eb0 con 0x7f90b00b4d60 2024-03-25T19:29:17.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.959+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851791679 0 0) 0x7f90b01381a0 con 0x7f90b014be90 2024-03-25T19:29:17.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.959+0000 7f90ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:17.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.959+0000 7f90ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:17.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.959+0000 7f90ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:17.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.959+0000 7f90ad7fa640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:29:17.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.959+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b0143900 con 0x7f90b014be90 2024-03-25T19:29:17.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.965+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640887572 0 0) 0x7f90b01dadd0 con 0x7f90b00b1730 2024-03-25T19:29:17.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.965+0000 7f90ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:17.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.965+0000 7f90ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:17.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.965+0000 7f90ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:17.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.965+0000 7f90ad7fa640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:29:17.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.965+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b01381a0 con 0x7f90b00b1730 2024-03-25T19:29:17.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.974+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2683706048 0 0) 0x7f9094004eb0 con 0x7f90b00b4d60 2024-03-25T19:29:17.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.974+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90b01dadd0 con 0x7f90b00b4d60 2024-03-25T19:29:17.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.974+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f909c001b70 con 0x7f90b00b4d60 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.974+0000 7f90ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.974+0000 7f90ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.974+0000 7f90ad7fa640 10 monclient(hunting): dump: 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:17.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.974+0000 7f90ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:17.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.980+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 58359368 0 0) 0x7f90b0143900 con 0x7f90b014be90 2024-03-25T19:29:17.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.980+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9094004eb0 con 0x7f90b014be90 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.980+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f90a8001a70 con 0x7f90b014be90 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.980+0000 7f90ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.980+0000 7f90ad7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.980+0000 7f90ad7fa640 10 monclient(hunting): dump: 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:17.980 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2298470263 0 0) 0x7f90b01dadd0 con 0x7f90b00b4d60 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 >> v1:172.21.15.179:6789/0 conn(0x7f90b014be90 legacy=0x7f90b01d8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 >> v1:172.21.15.52:6789/0 conn(0x7f90b00b1730 legacy=0x7f90b01db260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90ad7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90ad7fa640 1 monclient: found mon.c 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90ad7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b0143a90 con 0x7f90b00b4d60 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90ad7fa640 10 monclient: _finish_auth 0 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90ad7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90b7089640 5 monclient: authenticate success, global_id 4298 2024-03-25T19:29:17.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90b7089640 10 monclient: _renew_subs 2024-03-25T19:29:17.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90b7089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:17.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90b7089640 1 -- 172.21.15.52:0/2681468544 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b01db970 con 0x7f90b00b4d60 2024-03-25T19:29:17.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90b7089640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:17.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90b7089640 10 monclient: _renew_subs 2024-03-25T19:29:17.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90b7089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:17.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90b7089640 1 -- 172.21.15.52:0/2681468544 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b01e3430 con 0x7f90b00b4d60 2024-03-25T19:29:17.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.988+0000 7f90b7089640 1 librados: init done 2024-03-25T19:29:17.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.989+0000 7f90b7089640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:17.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.989+0000 7f90b7089640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:17.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.989+0000 7f90b7089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:17.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:17.989+0000 7f90b7089640 1 -- 172.21.15.52:0/2681468544 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b01401f0 con 0x7f90b00b4d60 2024-03-25T19:29:18.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.320+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f909c004bf0 con 0x7f90b00b4d60 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.320+0000 7f90ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.320+0000 7f90aeffd640 4 set_mon_vals no callback set 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.320+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f909c004f00 con 0x7f90b00b4d60 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.320+0000 7f90ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.320+0000 7f90ad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.320+0000 7f90ad7fa640 10 monclient: dump: 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:18.320 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:18.321 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:18.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.320+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f909c01aed0 con 0x7f90b00b4d60 2024-03-25T19:29:18.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.321+0000 7f90b4e25640 10 monclient: get_auth_request con 0x7f909406df10 auth_method 0 2024-03-25T19:29:18.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.321+0000 7f90ad7fa640 10 client.4298.objecter ms_handle_connect 0x7f909406df10 2024-03-25T19:29:18.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.327+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f909c054970 con 0x7f90b00b4d60 2024-03-25T19:29:18.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.327+0000 7f90ad7fa640 10 client.4298.objecter ms_dispatch 0x7f90b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:18.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.327+0000 7f90ad7fa640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:18.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.327+0000 7f90ad7fa640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:18.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.327+0000 7f90ad7fa640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-25T19:29:18.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.327+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f909c050480 con 0x7f90b00b4d60 2024-03-25T19:29:18.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.327+0000 7f90ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:18.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.327+0000 7f90ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:18.590 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.591+0000 7f90b7089640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:18.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.591+0000 7f90b7089640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:18.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.591+0000 7f90b7089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:18.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.591+0000 7f90b7089640 1 -- 172.21.15.52:0/2681468544 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90b01daf80 con 0x7f90b00b4d60 2024-03-25T19:29:18.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.623+0000 7f90ad7fa640 1 -- 172.21.15.52:0/2681468544 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f90b01daf80 con 0x7f90b00b4d60 2024-03-25T19:29:18.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.623+0000 7f90ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:18.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.623+0000 7f90ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:18.623 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:29:18.623 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:18.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 10 librados: watch_flush enter 2024-03-25T19:29:18.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 10 librados: watch_flush exit 2024-03-25T19:29:18.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-25T19:29:18.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 10 client.4298.objecter successfully canceled tick 2024-03-25T19:29:18.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 1 -- 172.21.15.52:0/2681468544 >> v1:172.21.15.52:6800/3967427147 conn(0x7f909406df10 legacy=0x7f90940703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 10 monclient: shutdown 2024-03-25T19:29:18.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:18.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 1 -- 172.21.15.52:0/2681468544 >> v1:172.21.15.52:6790/0 conn(0x7f90b00b4d60 legacy=0x7f90b01d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:18.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 1 -- 172.21.15.52:0/2681468544 shutdown_connections 2024-03-25T19:29:18.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 1 -- 172.21.15.52:0/2681468544 >> 172.21.15.52:0/2681468544 conn(0x7f90b00617a0 msgr2=0x7f90b00b6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:18.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 1 -- 172.21.15.52:0/2681468544 shutdown_connections 2024-03-25T19:29:18.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 1 -- 172.21.15.52:0/2681468544 wait complete. 2024-03-25T19:29:18.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:18.625+0000 7f90b7089640 1 librados: shutdown 2024-03-25T19:29:18.637 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:25.444 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:25.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.541+0000 7f6c3ce55640 10 monclient: get_monmap_and_config 2024-03-25T19:29:25.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.541+0000 7f6c3ce55640 10 monclient: build_initial_monmap 2024-03-25T19:29:25.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.541+0000 7f6c3ce55640 10 monclient: monmap: 2024-03-25T19:29:25.542 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:25.542 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:25.542 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:29:25.543280+0000 2024-03-25T19:29:25.543 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:29:25.543280+0000 2024-03-25T19:29:25.543 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:25.543 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:25.543 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:25.543 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:29:25.543 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:29:25.543 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:25.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.542+0000 7f6c3ce55640 1 Processor -- start 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.542+0000 7f6c3ce55640 1 -- start start 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.542+0000 7f6c3ce55640 10 monclient: init 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.543+0000 7f6c3ce55640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.543+0000 7f6c3ce55640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.543+0000 7f6c3ce55640 10 monclient(hunting): picked mon.noname-a con 0x7f6c380a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.543+0000 7f6c3ce55640 10 monclient(hunting): picked mon.noname-c con 0x7f6c3813f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.543+0000 7f6c3ce55640 10 monclient(hunting): picked mon.noname-b con 0x7f6c381430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.543+0000 7f6c3ce55640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c38142520 con 0x7f6c380a98f0 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.543+0000 7f6c3ce55640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c380b28a0 con 0x7f6c3813f710 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.543+0000 7f6c3ce55640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c380b2a30 con 0x7f6c381430c0 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.543+0000 7f6c3ce55640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:25.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.543+0000 7f6c3ce55640 10 monclient(hunting): authenticate will time out at 6013.724716s 2024-03-25T19:29:25.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.553+0000 7f6c3659c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f6c380a98f0 0x7f6c380a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:40532/0 (socket says 172.21.15.52:40532) 2024-03-25T19:29:25.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.553+0000 7f6c3659c640 1 -- 172.21.15.52:0/776153084 learned_addr learned my addr 172.21.15.52:0/776153084 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:25.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.567+0000 7f6c35d9b640 1 -- 172.21.15.52:0/776153084 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072100841 0 0) 0x7f6c380b28a0 con 0x7f6c3813f710 2024-03-25T19:29:25.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.567+0000 7f6c35d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.567+0000 7f6c35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.567+0000 7f6c35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.567+0000 7f6c35d9b640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.567+0000 7f6c35d9b640 1 -- 172.21.15.52:0/776153084 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c24003520 con 0x7f6c3813f710 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 1 -- 172.21.15.52:0/776153084 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460251485 0 0) 0x7f6c38142520 con 0x7f6c380a98f0 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 1 -- 172.21.15.52:0/776153084 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c380b28a0 con 0x7f6c380a98f0 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 1 -- 172.21.15.52:0/776153084 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797516264 0 0) 0x7f6c24003520 con 0x7f6c3813f710 2024-03-25T19:29:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 1 -- 172.21.15.52:0/776153084 >> v1:172.21.15.52:6790/0 conn(0x7f6c381430c0 legacy=0x7f6c3814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 1 -- 172.21.15.52:0/776153084 >> v1:172.21.15.52:6789/0 conn(0x7f6c380a98f0 legacy=0x7f6c380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:29:25.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 1 monclient: found mon.noname-c 2024-03-25T19:29:25.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:25.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:29:25.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 1 -- 172.21.15.52:0/776153084 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c380b44c0 con 0x7f6c3813f710 2024-03-25T19:29:25.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 10 monclient: _finish_auth 0 2024-03-25T19:29:25.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:29:25.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c35d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c3ce55640 5 monclient: authenticate success, global_id 4312 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.573+0000 7f6c3ce55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c35d9b640 1 -- 172.21.15.52:0/776153084 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6c20001650 con 0x7f6c3813f710 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c35d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c35d9b640 10 monclient: dump: 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:25.707 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:25.708 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:25.708 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:25.708 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:25.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c35d9b640 1 -- 172.21.15.52:0/776153084 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c200018a0 con 0x7f6c3813f710 2024-03-25T19:29:25.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c35d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:25.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c35d9b640 1 -- 172.21.15.52:0/776153084 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6c20001c30 con 0x7f6c3813f710 2024-03-25T19:29:25.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:25.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c35d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:25.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c35d9b640 10 monclient: dump: 2024-03-25T19:29:25.708 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:25.708 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c3ce55640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c3ce55640 4 set_mon_vals no callback set 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c3ce55640 10 monclient: shutdown 2024-03-25T19:29:25.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c3ce55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.583+0000 7f6c3ce55640 1 -- 172.21.15.52:0/776153084 >> v1:172.21.15.179:6789/0 conn(0x7f6c3813f710 legacy=0x7f6c381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.584+0000 7f6c3ce55640 1 -- 172.21.15.52:0/776153084 shutdown_connections 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.584+0000 7f6c3ce55640 1 -- 172.21.15.52:0/776153084 >> 172.21.15.52:0/776153084 conn(0x7f6c380b7ea0 msgr2=0x7f6c380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.584+0000 7f6c3ce55640 1 -- 172.21.15.52:0/776153084 shutdown_connections 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.584+0000 7f6c3ce55640 1 -- 172.21.15.52:0/776153084 wait complete. 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.585+0000 7f6c3ce55640 10 monclient: build_initial_monmap 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.585+0000 7f6c3ce55640 10 monclient: monmap: 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:29:25.710 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.585+0000 7f6c3ce55640 1 librados: starting msgr at 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.585+0000 7f6c3ce55640 1 librados: starting objecter 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.585+0000 7f6c3ce55640 1 Processor -- start 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.585+0000 7f6c3ce55640 1 -- start start 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.585+0000 7f6c3ce55640 1 librados: setting wanted keys 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.585+0000 7f6c3ce55640 1 librados: calling monclient init 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.585+0000 7f6c3ce55640 10 monclient: init 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.586+0000 7f6c3ce55640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.586+0000 7f6c3ce55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:25.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.586+0000 7f6c3ce55640 10 monclient(hunting): picked mon.noname-b con 0x7f6c380a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.586+0000 7f6c3ce55640 10 monclient(hunting): picked mon.noname-c con 0x7f6c3813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.586+0000 7f6c3ce55640 10 monclient(hunting): picked mon.noname-a con 0x7f6c381430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.586+0000 7f6c3ce55640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c380b7ba0 con 0x7f6c381430c0 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.586+0000 7f6c3ce55640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c380b78c0 con 0x7f6c380a98f0 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.586+0000 7f6c3ce55640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c38136c30 con 0x7f6c3813f710 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.586+0000 7f6c3ce55640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.586+0000 7f6c3ce55640 10 monclient(hunting): authenticate will time out at 6013.767545s 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.597+0000 7f6c2ffff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f6c3813f710 0x7f6c381375a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:56006/0 (socket says 172.21.15.52:56006) 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.597+0000 7f6c2ffff640 1 -- 172.21.15.52:0/1282644843 learned_addr learned my addr 172.21.15.52:0/1282644843 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.599+0000 7f6c2effd640 10 client.?.objecter ms_handle_connect 0x7f6c3813f710 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.599+0000 7f6c2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:25.712 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.606+0000 7f6c2effd640 10 client.?.objecter ms_handle_connect 0x7f6c380a98f0 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.606+0000 7f6c2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.613+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435568892 0 0) 0x7f6c38136c30 con 0x7f6c3813f710 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.614+0000 7f6c2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.614+0000 7f6c2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.614+0000 7f6c2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.614+0000 7f6c2effd640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.614+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c18004eb0 con 0x7f6c3813f710 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.619+0000 7f6c2effd640 10 client.?.objecter ms_handle_connect 0x7f6c381430c0 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.619+0000 7f6c2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.621+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320754297 0 0) 0x7f6c380b78c0 con 0x7f6c380a98f0 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.621+0000 7f6c2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.621+0000 7f6c2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.621+0000 7f6c2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.621+0000 7f6c2effd640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.621+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c38136c30 con 0x7f6c380a98f0 2024-03-25T19:29:25.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.627+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154602595 0 0) 0x7f6c380b7ba0 con 0x7f6c381430c0 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.627+0000 7f6c2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.627+0000 7f6c2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.627+0000 7f6c2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.627+0000 7f6c2effd640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.627+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c380b78c0 con 0x7f6c381430c0 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.628+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3402253189 0 0) 0x7f6c18004eb0 con 0x7f6c3813f710 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.628+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c380b7ba0 con 0x7f6c3813f710 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.628+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6c20001c70 con 0x7f6c3813f710 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.628+0000 7f6c2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.628+0000 7f6c2effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.628+0000 7f6c2effd640 10 monclient(hunting): dump: 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:25.714 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.628+0000 7f6c2effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.648+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4141594128 0 0) 0x7f6c38136c30 con 0x7f6c380a98f0 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.648+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c18004eb0 con 0x7f6c380a98f0 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.648+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6c28001930 con 0x7f6c380a98f0 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.648+0000 7f6c2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.648+0000 7f6c2effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.648+0000 7f6c2effd640 10 monclient(hunting): dump: 2024-03-25T19:29:25.715 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 158511902 0 0) 0x7f6c380b7ba0 con 0x7f6c3813f710 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 >> v1:172.21.15.179:6789/0 conn(0x7f6c380a98f0 legacy=0x7f6c3813b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 >> v1:172.21.15.52:6789/0 conn(0x7f6c381430c0 legacy=0x7f6c381e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c2effd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c2effd640 1 monclient: found mon.c 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:25.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c2effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c38136ea0 con 0x7f6c3813f710 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c2effd640 10 monclient: _finish_auth 0 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c2effd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c3ce55640 5 monclient: authenticate success, global_id 4304 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c3ce55640 10 monclient: _renew_subs 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c3ce55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c3ce55640 1 -- 172.21.15.52:0/1282644843 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c38137180 con 0x7f6c3813f710 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c3ce55640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c3ce55640 10 monclient: _renew_subs 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c3ce55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c3ce55640 1 -- 172.21.15.52:0/1282644843 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c38136070 con 0x7f6c3813f710 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.650+0000 7f6c3ce55640 1 librados: init done 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.651+0000 7f6c3ce55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:25.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.651+0000 7f6c3ce55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:25.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.651+0000 7f6c3ce55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:25.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.651+0000 7f6c3ce55640 1 -- 172.21.15.52:0/1282644843 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c380627b0 con 0x7f6c3813f710 2024-03-25T19:29:25.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.970+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c20001230 con 0x7f6c3813f710 2024-03-25T19:29:25.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.970+0000 7f6c2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:25.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.970+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6c20003bc0 con 0x7f6c3813f710 2024-03-25T19:29:25.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.970+0000 7f6c2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:25.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.970+0000 7f6c2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:25.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.970+0000 7f6c2effd640 10 monclient: dump: 2024-03-25T19:29:25.971 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:25.971 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:25.971 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:25.971 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:25.972 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:25.972 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:25.972 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:25.972 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:25.972 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:25.972 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:25.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.970+0000 7f6c3559a640 4 set_mon_vals no callback set 2024-03-25T19:29:25.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.971+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f6c20003de0 con 0x7f6c3813f710 2024-03-25T19:29:25.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.971+0000 7f6c3659c640 10 monclient: get_auth_request con 0x7f6c1806df20 auth_method 0 2024-03-25T19:29:25.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.972+0000 7f6c2effd640 10 client.4304.objecter ms_handle_connect 0x7f6c1806df20 2024-03-25T19:29:25.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.977+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f6c200549a0 con 0x7f6c3813f710 2024-03-25T19:29:25.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.977+0000 7f6c2effd640 10 client.4304.objecter ms_dispatch 0x7f6c38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:25.978 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.977+0000 7f6c2effd640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:25.978 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.977+0000 7f6c2effd640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:25.978 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.977+0000 7f6c2effd640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-25T19:29:25.978 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.977+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c200504b0 con 0x7f6c3813f710 2024-03-25T19:29:25.978 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.977+0000 7f6c2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:25.978 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:25.977+0000 7f6c2effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:26.109 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:02:19.708 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:26.120 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:02:19.719 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:26.150 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:02:19.749 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:26.169 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:02:19.711 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:26.174 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:02:19.716 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:26.175 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:02:19.774 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:26.182 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:02:19.781 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:26.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.191+0000 7f6c3ce55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:26.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.191+0000 7f6c3ce55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:26.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.191+0000 7f6c3ce55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:26.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.191+0000 7f6c3ce55640 1 -- 172.21.15.52:0/1282644843 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c3813c790 con 0x7f6c3813f710 2024-03-25T19:29:26.197 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:02:19.739 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:26.223 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:02:19.822 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:26.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.267+0000 7f6c2effd640 1 -- 172.21.15.52:0/1282644843 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f6c200523e0 con 0x7f6c3813f710 2024-03-25T19:29:26.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.267+0000 7f6c2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:26.268 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.267+0000 7f6c2effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:26.268 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:29:26.268 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:26.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 10 librados: watch_flush enter 2024-03-25T19:29:26.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 10 librados: watch_flush exit 2024-03-25T19:29:26.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-25T19:29:26.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 10 client.4304.objecter successfully canceled tick 2024-03-25T19:29:26.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 1 -- 172.21.15.52:0/1282644843 >> v1:172.21.15.52:6800/3967427147 conn(0x7f6c1806df20 legacy=0x7f6c180703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:26.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 10 monclient: shutdown 2024-03-25T19:29:26.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:26.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 1 -- 172.21.15.52:0/1282644843 >> v1:172.21.15.52:6790/0 conn(0x7f6c3813f710 legacy=0x7f6c381375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:26.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 1 -- 172.21.15.52:0/1282644843 shutdown_connections 2024-03-25T19:29:26.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 1 -- 172.21.15.52:0/1282644843 >> 172.21.15.52:0/1282644843 conn(0x7f6c380b7ea0 msgr2=0x7f6c38074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:26.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 1 -- 172.21.15.52:0/1282644843 shutdown_connections 2024-03-25T19:29:26.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 1 -- 172.21.15.52:0/1282644843 wait complete. 2024-03-25T19:29:26.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:26.269+0000 7f6c3ce55640 1 librados: shutdown 2024-03-25T19:29:26.282 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:33.088 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:33.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.200+0000 7fc7acbef640 10 monclient: get_monmap_and_config 2024-03-25T19:29:33.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.200+0000 7fc7acbef640 10 monclient: build_initial_monmap 2024-03-25T19:29:33.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.200+0000 7fc7acbef640 10 monclient: monmap: 2024-03-25T19:29:33.201 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:33.201 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:33.202 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:29:33.202469+0000 2024-03-25T19:29:33.202 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:29:33.202469+0000 2024-03-25T19:29:33.202 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:33.202 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:33.202 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:33.202 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:29:33.202 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:29:33.202 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:33.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.201+0000 7fc7acbef640 1 Processor -- start 2024-03-25T19:29:33.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.201+0000 7fc7acbef640 1 -- start start 2024-03-25T19:29:33.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.201+0000 7fc7acbef640 10 monclient: init 2024-03-25T19:29:33.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.202+0000 7fc7acbef640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:33.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.202+0000 7fc7acbef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:33.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.202+0000 7fc7acbef640 10 monclient(hunting): picked mon.noname-a con 0x7fc7a80a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:33.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.202+0000 7fc7acbef640 10 monclient(hunting): picked mon.noname-c con 0x7fc7a813f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:33.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.202+0000 7fc7acbef640 10 monclient(hunting): picked mon.noname-b con 0x7fc7a81430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:33.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.202+0000 7fc7acbef640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a8142520 con 0x7fc7a80a98f0 2024-03-25T19:29:33.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.202+0000 7fc7acbef640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a80b28a0 con 0x7fc7a813f710 2024-03-25T19:29:33.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.202+0000 7fc7acbef640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a80b2a30 con 0x7fc7a81430c0 2024-03-25T19:29:33.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.202+0000 7fc7acbef640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:33.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.202+0000 7fc7acbef640 10 monclient(hunting): authenticate will time out at 6021.383646s 2024-03-25T19:29:33.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.212+0000 7fc7a5d9b640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc7a813f710 0x7fc7a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:59974/0 (socket says 172.21.15.52:59974) 2024-03-25T19:29:33.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.212+0000 7fc7a5d9b640 1 -- 172.21.15.52:0/4173893958 learned_addr learned my addr 172.21.15.52:0/4173893958 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:33.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.232+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71277855 0 0) 0x7fc7a80b28a0 con 0x7fc7a813f710 2024-03-25T19:29:33.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.232+0000 7fc7a559a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.232+0000 7fc7a559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.232+0000 7fc7a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.232+0000 7fc7a559a640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:29:33.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.232+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc790003520 con 0x7fc7a813f710 2024-03-25T19:29:33.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.233+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593668936 0 0) 0x7fc7a8142520 con 0x7fc7a80a98f0 2024-03-25T19:29:33.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.233+0000 7fc7a559a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.233+0000 7fc7a559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.233+0000 7fc7a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.233+0000 7fc7a559a640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:29:33.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.233+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a80b28a0 con 0x7fc7a80a98f0 2024-03-25T19:29:33.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.242+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225755071 0 0) 0x7fc7a80b2a30 con 0x7fc7a81430c0 2024-03-25T19:29:33.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.242+0000 7fc7a559a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.242+0000 7fc7a559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.242+0000 7fc7a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.242+0000 7fc7a559a640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:29:33.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.242+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a8142520 con 0x7fc7a81430c0 2024-03-25T19:29:33.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2073044723 0 0) 0x7fc790003520 con 0x7fc7a813f710 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 >> v1:172.21.15.52:6790/0 conn(0x7fc7a81430c0 legacy=0x7fc7a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 >> v1:172.21.15.52:6789/0 conn(0x7fc7a80a98f0 legacy=0x7fc7a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 10 monclient: _finish_hunting 0 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 1 monclient: found mon.noname-c 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a80b44c0 con 0x7fc7a813f710 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 10 monclient: _finish_auth 0 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 10 monclient: _check_auth_tickets 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc79c001690 con 0x7fc7a813f710 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7a559a640 10 monclient: dump: 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7acbef640 5 monclient: authenticate success, global_id 4318 2024-03-25T19:29:33.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.244+0000 7fc7acbef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:33.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.257+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc79c0017f0 con 0x7fc7a813f710 2024-03-25T19:29:33.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.257+0000 7fc7a559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:33.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.257+0000 7fc7a559a640 1 -- 172.21.15.52:0/4173893958 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc79c001c00 con 0x7fc7a813f710 2024-03-25T19:29:33.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.257+0000 7fc7a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.257+0000 7fc7a559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:33.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.257+0000 7fc7a559a640 10 monclient: dump: 2024-03-25T19:29:33.257 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:33.258 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:33.258 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:33.258 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:33.258 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.258 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:33.258 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:33.258 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:33.258 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:33.258 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:33.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.257+0000 7fc7acbef640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:33.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.257+0000 7fc7acbef640 4 set_mon_vals no callback set 2024-03-25T19:29:33.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.257+0000 7fc7acbef640 10 monclient: shutdown 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.257+0000 7fc7acbef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.257+0000 7fc7acbef640 1 -- 172.21.15.52:0/4173893958 >> v1:172.21.15.179:6789/0 conn(0x7fc7a813f710 legacy=0x7fc7a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.258+0000 7fc7acbef640 1 -- 172.21.15.52:0/4173893958 shutdown_connections 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.258+0000 7fc7acbef640 1 -- 172.21.15.52:0/4173893958 >> 172.21.15.52:0/4173893958 conn(0x7fc7a80b7ea0 msgr2=0x7fc7a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.258+0000 7fc7acbef640 1 -- 172.21.15.52:0/4173893958 shutdown_connections 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.258+0000 7fc7acbef640 1 -- 172.21.15.52:0/4173893958 wait complete. 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.259+0000 7fc7acbef640 10 monclient: build_initial_monmap 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.259+0000 7fc7acbef640 10 monclient: monmap: 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:29:33.260 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:33.261 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:29:33.261 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:29:33.261 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:33.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.259+0000 7fc7acbef640 1 librados: starting msgr at 2024-03-25T19:29:33.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.259+0000 7fc7acbef640 1 librados: starting objecter 2024-03-25T19:29:33.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.259+0000 7fc7acbef640 1 Processor -- start 2024-03-25T19:29:33.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.259+0000 7fc7acbef640 1 -- start start 2024-03-25T19:29:33.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.259+0000 7fc7acbef640 1 librados: setting wanted keys 2024-03-25T19:29:33.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.259+0000 7fc7acbef640 1 librados: calling monclient init 2024-03-25T19:29:33.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.259+0000 7fc7acbef640 10 monclient: init 2024-03-25T19:29:33.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.260+0000 7fc7acbef640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:33.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.260+0000 7fc7acbef640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:33.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.260+0000 7fc7acbef640 10 monclient(hunting): picked mon.noname-b con 0x7fc7a80a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:33.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.260+0000 7fc7acbef640 10 monclient(hunting): picked mon.noname-c con 0x7fc7a813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:33.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.260+0000 7fc7acbef640 10 monclient(hunting): picked mon.noname-a con 0x7fc7a81430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:33.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.260+0000 7fc7acbef640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a8138c10 con 0x7fc7a81430c0 2024-03-25T19:29:33.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.260+0000 7fc7acbef640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a8138e60 con 0x7fc7a80a98f0 2024-03-25T19:29:33.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.260+0000 7fc7acbef640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a8139140 con 0x7fc7a813f710 2024-03-25T19:29:33.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.260+0000 7fc7acbef640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:33.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.260+0000 7fc7acbef640 10 monclient(hunting): authenticate will time out at 6021.441511s 2024-03-25T19:29:33.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.271+0000 7fc7a659c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc7a80a98f0 0x7fc7a8137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:60024/0 (socket says 172.21.15.52:60024) 2024-03-25T19:29:33.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.271+0000 7fc7a659c640 1 -- 172.21.15.52:0/4027022804 learned_addr learned my addr 172.21.15.52:0/4027022804 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:33.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.273+0000 7fc796ffd640 10 client.?.objecter ms_handle_connect 0x7fc7a80a98f0 2024-03-25T19:29:33.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.274+0000 7fc796ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:33.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.278+0000 7fc796ffd640 10 client.?.objecter ms_handle_connect 0x7fc7a81430c0 2024-03-25T19:29:33.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.278+0000 7fc796ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:33.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.279+0000 7fc796ffd640 10 client.?.objecter ms_handle_connect 0x7fc7a813f710 2024-03-25T19:29:33.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.279+0000 7fc796ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:33.285 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.284+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542330949 0 0) 0x7fc7a8138e60 con 0x7fc7a80a98f0 2024-03-25T19:29:33.285 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.284+0000 7fc796ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.285 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.284+0000 7fc796ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.285 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.284+0000 7fc796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.285 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.284+0000 7fc796ffd640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:29:33.285 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.284+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc788004eb0 con 0x7fc7a80a98f0 2024-03-25T19:29:33.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.290+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572834340 0 0) 0x7fc7a8138c10 con 0x7fc7a81430c0 2024-03-25T19:29:33.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.290+0000 7fc796ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.290+0000 7fc796ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.290+0000 7fc796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.290+0000 7fc796ffd640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:29:33.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.290+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a8138e60 con 0x7fc7a81430c0 2024-03-25T19:29:33.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.294+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976308674 0 0) 0x7fc7a8139140 con 0x7fc7a813f710 2024-03-25T19:29:33.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.294+0000 7fc796ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:33.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.294+0000 7fc796ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:33.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.294+0000 7fc796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:33.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.294+0000 7fc796ffd640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:29:33.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.294+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a8138c10 con 0x7fc7a813f710 2024-03-25T19:29:33.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.299+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4264398804 0 0) 0x7fc788004eb0 con 0x7fc7a80a98f0 2024-03-25T19:29:33.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.299+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7a8139140 con 0x7fc7a80a98f0 2024-03-25T19:29:33.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.299+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc798001a30 con 0x7fc7a80a98f0 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.299+0000 7fc796ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.299+0000 7fc796ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.299+0000 7fc796ffd640 10 monclient(hunting): dump: 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:33.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.299+0000 7fc796ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.304+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3154628339 0 0) 0x7fc7a8138e60 con 0x7fc7a81430c0 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.304+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc788004eb0 con 0x7fc7a81430c0 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.304+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc7a0001b50 con 0x7fc7a81430c0 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.304+0000 7fc796ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.304+0000 7fc796ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.304+0000 7fc796ffd640 10 monclient(hunting): dump: 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.305 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:33.306 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:33.306 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:33.306 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:33.306 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2620355862 0 0) 0x7fc7a8139140 con 0x7fc7a80a98f0 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 >> v1:172.21.15.52:6790/0 conn(0x7fc7a813f710 legacy=0x7fc7a8132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 >> v1:172.21.15.52:6789/0 conn(0x7fc7a81430c0 legacy=0x7fc7a81e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc796ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc796ffd640 1 monclient: found mon.b 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc796ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc796ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a8139920 con 0x7fc7a80a98f0 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc796ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc796ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc796ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:33.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc7acbef640 5 monclient: authenticate success, global_id 4321 2024-03-25T19:29:33.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc7acbef640 10 monclient: _renew_subs 2024-03-25T19:29:33.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc7acbef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:33.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc7acbef640 1 -- 172.21.15.52:0/4027022804 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a81382e0 con 0x7fc7a80a98f0 2024-03-25T19:29:33.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc7acbef640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:33.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc7acbef640 10 monclient: _renew_subs 2024-03-25T19:29:33.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc7acbef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:33.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc7acbef640 1 -- 172.21.15.52:0/4027022804 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a80ab8b0 con 0x7fc7a80a98f0 2024-03-25T19:29:33.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.311+0000 7fc7acbef640 1 librados: init done 2024-03-25T19:29:33.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.312+0000 7fc7acbef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:33.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.312+0000 7fc7acbef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:33.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.312+0000 7fc7acbef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:33.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.312+0000 7fc7acbef640 1 -- 172.21.15.52:0/4027022804 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a8137a20 con 0x7fc7a80a98f0 2024-03-25T19:29:33.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.605+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc798001c70 con 0x7fc7a80a98f0 2024-03-25T19:29:33.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.605+0000 7fc796ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:33.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.605+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc798003a60 con 0x7fc7a80a98f0 2024-03-25T19:29:33.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.605+0000 7fc796ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:33.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.605+0000 7fc796ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:33.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.605+0000 7fc796ffd640 10 monclient: dump: 2024-03-25T19:29:33.606 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:33.606 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:33.607 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:33.607 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:33.607 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:33.607 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:33.607 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:33.607 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:33.607 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:33.607 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:33.607 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.605+0000 7fc7a4d99640 4 set_mon_vals no callback set 2024-03-25T19:29:33.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.606+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fc798003c40 con 0x7fc7a80a98f0 2024-03-25T19:29:33.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.607+0000 7fc7a5d9b640 10 monclient: get_auth_request con 0x7fc78806df70 auth_method 0 2024-03-25T19:29:33.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.607+0000 7fc796ffd640 10 client.4321.objecter ms_handle_connect 0x7fc78806df70 2024-03-25T19:29:33.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.611+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fc7980548d0 con 0x7fc7a80a98f0 2024-03-25T19:29:33.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.611+0000 7fc796ffd640 10 client.4321.objecter ms_dispatch 0x7fc7a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:33.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.611+0000 7fc796ffd640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:33.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.611+0000 7fc796ffd640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:33.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.611+0000 7fc796ffd640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T19:29:33.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.611+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc7980503e0 con 0x7fc7a80a98f0 2024-03-25T19:29:33.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.611+0000 7fc796ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:33.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.611+0000 7fc796ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:33.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.848+0000 7fc7acbef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:33.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.848+0000 7fc7acbef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:33.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.848+0000 7fc7acbef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:33.849 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.848+0000 7fc7acbef640 1 -- 172.21.15.52:0/4027022804 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7a813d010 con 0x7fc7a80a98f0 2024-03-25T19:29:33.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.890+0000 7fc796ffd640 1 -- 172.21.15.52:0/4027022804 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fc798052310 con 0x7fc7a80a98f0 2024-03-25T19:29:33.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.890+0000 7fc796ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:33.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.890+0000 7fc796ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:33.891 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:29:33.891 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:33.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.892+0000 7fc7acbef640 10 librados: watch_flush enter 2024-03-25T19:29:33.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.892+0000 7fc7acbef640 10 librados: watch_flush exit 2024-03-25T19:29:33.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.892+0000 7fc7acbef640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T19:29:33.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.892+0000 7fc7acbef640 10 client.4321.objecter successfully canceled tick 2024-03-25T19:29:33.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.892+0000 7fc7acbef640 1 -- 172.21.15.52:0/4027022804 >> v1:172.21.15.52:6800/3967427147 conn(0x7fc78806df70 legacy=0x7fc788070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.892+0000 7fc7acbef640 10 monclient: shutdown 2024-03-25T19:29:33.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.892+0000 7fc7acbef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:33.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.892+0000 7fc7acbef640 1 -- 172.21.15.52:0/4027022804 >> v1:172.21.15.179:6789/0 conn(0x7fc7a80a98f0 legacy=0x7fc7a8137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:33.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.893+0000 7fc7acbef640 1 -- 172.21.15.52:0/4027022804 shutdown_connections 2024-03-25T19:29:33.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.893+0000 7fc7acbef640 1 -- 172.21.15.52:0/4027022804 >> 172.21.15.52:0/4027022804 conn(0x7fc7a80b7ea0 msgr2=0x7fc7a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:33.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.893+0000 7fc7acbef640 1 -- 172.21.15.52:0/4027022804 shutdown_connections 2024-03-25T19:29:33.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.893+0000 7fc7acbef640 1 -- 172.21.15.52:0/4027022804 wait complete. 2024-03-25T19:29:33.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:33.893+0000 7fc7acbef640 1 librados: shutdown 2024-03-25T19:29:33.904 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:40.711 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:40.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.826+0000 7f1b4c7c6640 10 monclient: get_monmap_and_config 2024-03-25T19:29:40.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.826+0000 7f1b4c7c6640 10 monclient: build_initial_monmap 2024-03-25T19:29:40.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.827+0000 7f1b4c7c6640 10 monclient: monmap: 2024-03-25T19:29:40.828 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:40.828 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:40.828 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:29:40.828562+0000 2024-03-25T19:29:40.828 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:29:40.828562+0000 2024-03-25T19:29:40.828 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:40.828 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:40.828 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:40.828 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:29:40.828 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:29:40.828 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:40.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.827+0000 7f1b4c7c6640 1 Processor -- start 2024-03-25T19:29:40.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.827+0000 7f1b4c7c6640 1 -- start start 2024-03-25T19:29:40.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.827+0000 7f1b4c7c6640 10 monclient: init 2024-03-25T19:29:40.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.828+0000 7f1b4c7c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:40.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.828+0000 7f1b4c7c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:40.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.828+0000 7f1b4c7c6640 10 monclient(hunting): picked mon.noname-b con 0x7f1b440a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:40.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.828+0000 7f1b4c7c6640 10 monclient(hunting): picked mon.noname-c con 0x7f1b4413f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:40.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.828+0000 7f1b4c7c6640 10 monclient(hunting): picked mon.noname-a con 0x7f1b441430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:40.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.828+0000 7f1b4c7c6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b44142520 con 0x7f1b441430c0 2024-03-25T19:29:40.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.828+0000 7f1b4c7c6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440b28a0 con 0x7f1b4413f710 2024-03-25T19:29:40.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.828+0000 7f1b4c7c6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440b2a30 con 0x7f1b440a98f0 2024-03-25T19:29:40.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.828+0000 7f1b4c7c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:40.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.828+0000 7f1b4c7c6640 10 monclient(hunting): authenticate will time out at 6029.009722s 2024-03-25T19:29:40.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.834+0000 7f1b49d61640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f1b4413f710 0x7f1b441429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:58180/0 (socket says 172.21.15.52:58180) 2024-03-25T19:29:40.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.834+0000 7f1b49d61640 1 -- 172.21.15.52:0/2229283302 learned_addr learned my addr 172.21.15.52:0/2229283302 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.850+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378181667 0 0) 0x7f1b440b2a30 con 0x7f1b440a98f0 2024-03-25T19:29:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.850+0000 7f1b49560640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.850+0000 7f1b49560640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.850+0000 7f1b49560640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.850+0000 7f1b49560640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:29:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.850+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b30003520 con 0x7f1b440a98f0 2024-03-25T19:29:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.851+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690791520 0 0) 0x7f1b440b28a0 con 0x7f1b4413f710 2024-03-25T19:29:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.851+0000 7f1b49560640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.851+0000 7f1b49560640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.851+0000 7f1b49560640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.851+0000 7f1b49560640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:29:40.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.851+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b440b2a30 con 0x7f1b4413f710 2024-03-25T19:29:40.861 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.861+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221288524 0 0) 0x7f1b44142520 con 0x7f1b441430c0 2024-03-25T19:29:40.861 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.861+0000 7f1b49560640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.861 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.861+0000 7f1b49560640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.861 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.861+0000 7f1b49560640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.861 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.861+0000 7f1b49560640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:29:40.861 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.861+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b440b28a0 con 0x7f1b441430c0 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3739322001 0 0) 0x7f1b440b2a30 con 0x7f1b4413f710 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 >> v1:172.21.15.52:6790/0 conn(0x7f1b440a98f0 legacy=0x7f1b440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 >> v1:172.21.15.52:6789/0 conn(0x7f1b441430c0 legacy=0x7f1b4414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 10 monclient: _finish_hunting 0 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 1 monclient: found mon.noname-c 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b440b44c0 con 0x7f1b4413f710 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 10 monclient: _finish_auth 0 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 10 monclient: _check_auth_tickets 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1b34001670 con 0x7f1b4413f710 2024-03-25T19:29:40.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b49560640 10 monclient: dump: 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b4c7c6640 5 monclient: authenticate success, global_id 4324 2024-03-25T19:29:40.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.864+0000 7f1b4c7c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:40.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.884+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b340017d0 con 0x7f1b4413f710 2024-03-25T19:29:40.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.884+0000 7f1b49560640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:40.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.884+0000 7f1b49560640 1 -- 172.21.15.52:0/2229283302 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1b34001be0 con 0x7f1b4413f710 2024-03-25T19:29:40.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.884+0000 7f1b49560640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.884+0000 7f1b49560640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.884+0000 7f1b49560640 10 monclient: dump: 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:40.885 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:40.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.884+0000 7f1b4c7c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:40.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.884+0000 7f1b4c7c6640 4 set_mon_vals no callback set 2024-03-25T19:29:40.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.885+0000 7f1b4c7c6640 10 monclient: shutdown 2024-03-25T19:29:40.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.885+0000 7f1b4c7c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:40.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.885+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/2229283302 >> v1:172.21.15.179:6789/0 conn(0x7f1b4413f710 legacy=0x7f1b441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.885+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/2229283302 shutdown_connections 2024-03-25T19:29:40.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.885+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/2229283302 >> 172.21.15.52:0/2229283302 conn(0x7f1b440b7ea0 msgr2=0x7f1b440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:40.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.885+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/2229283302 shutdown_connections 2024-03-25T19:29:40.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.885+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/2229283302 wait complete. 2024-03-25T19:29:40.887 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.886+0000 7f1b4c7c6640 10 monclient: build_initial_monmap 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.886+0000 7f1b4c7c6640 10 monclient: monmap: 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.886+0000 7f1b4c7c6640 1 librados: starting msgr at 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.886+0000 7f1b4c7c6640 1 librados: starting objecter 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.886+0000 7f1b4c7c6640 1 Processor -- start 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.886+0000 7f1b4c7c6640 1 -- start start 2024-03-25T19:29:40.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.886+0000 7f1b4c7c6640 1 librados: setting wanted keys 2024-03-25T19:29:40.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.886+0000 7f1b4c7c6640 1 librados: calling monclient init 2024-03-25T19:29:40.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.886+0000 7f1b4c7c6640 10 monclient: init 2024-03-25T19:29:40.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.887+0000 7f1b4c7c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:40.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.887+0000 7f1b4c7c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:40.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.887+0000 7f1b4c7c6640 10 monclient(hunting): picked mon.noname-c con 0x7f1b440a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:40.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.887+0000 7f1b4c7c6640 10 monclient(hunting): picked mon.noname-b con 0x7f1b4413f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:40.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.887+0000 7f1b4c7c6640 10 monclient(hunting): picked mon.noname-a con 0x7f1b441430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:40.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.887+0000 7f1b4c7c6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440abde0 con 0x7f1b441430c0 2024-03-25T19:29:40.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.887+0000 7f1b4c7c6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440ac0c0 con 0x7f1b4413f710 2024-03-25T19:29:40.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.887+0000 7f1b4c7c6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b44139250 con 0x7f1b440a98f0 2024-03-25T19:29:40.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.887+0000 7f1b4c7c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:40.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.887+0000 7f1b4c7c6640 10 monclient(hunting): authenticate will time out at 6029.068471s 2024-03-25T19:29:40.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.890+0000 7f1b4a562640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f1b440a98f0 0x7f1b44138980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:53586/0 (socket says 172.21.15.52:53586) 2024-03-25T19:29:40.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.890+0000 7f1b4a562640 1 -- 172.21.15.52:0/1044209363 learned_addr learned my addr 172.21.15.52:0/1044209363 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:40.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.891+0000 7f1b3affd640 10 client.?.objecter ms_handle_connect 0x7f1b440a98f0 2024-03-25T19:29:40.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.891+0000 7f1b3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:40.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.899+0000 7f1b3affd640 10 client.?.objecter ms_handle_connect 0x7f1b441430c0 2024-03-25T19:29:40.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.899+0000 7f1b3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:40.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.901+0000 7f1b3affd640 10 client.?.objecter ms_handle_connect 0x7f1b4413f710 2024-03-25T19:29:40.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.901+0000 7f1b3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:40.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.901+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260502897 0 0) 0x7f1b44139250 con 0x7f1b440a98f0 2024-03-25T19:29:40.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.901+0000 7f1b3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.901+0000 7f1b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.901+0000 7f1b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.901+0000 7f1b3affd640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:29:40.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.901+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b28004eb0 con 0x7f1b440a98f0 2024-03-25T19:29:40.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.913+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567628105 0 0) 0x7f1b440abde0 con 0x7f1b441430c0 2024-03-25T19:29:40.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.913+0000 7f1b3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.913+0000 7f1b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.913+0000 7f1b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.913+0000 7f1b3affd640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:29:40.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.913+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b44139250 con 0x7f1b441430c0 2024-03-25T19:29:40.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.920+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276203544 0 0) 0x7f1b440ac0c0 con 0x7f1b4413f710 2024-03-25T19:29:40.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.920+0000 7f1b3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:40.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.920+0000 7f1b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:40.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.920+0000 7f1b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:40.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.920+0000 7f1b3affd640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:29:40.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.920+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b440abde0 con 0x7f1b4413f710 2024-03-25T19:29:40.926 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.925+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3335647245 0 0) 0x7f1b28004eb0 con 0x7f1b440a98f0 2024-03-25T19:29:40.926 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.925+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b440ac0c0 con 0x7f1b440a98f0 2024-03-25T19:29:40.926 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.925+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1b40001990 con 0x7f1b440a98f0 2024-03-25T19:29:40.926 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.925+0000 7f1b3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:40.926 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.925+0000 7f1b3affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:40.926 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.925+0000 7f1b3affd640 10 monclient(hunting): dump: 2024-03-25T19:29:40.926 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:40.926 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:40.927 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:40.927 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:40.927 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:40.927 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:40.927 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:40.927 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:40.927 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:40.927 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:40.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.925+0000 7f1b3affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:40.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.941+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2063825458 0 0) 0x7f1b44139250 con 0x7f1b441430c0 2024-03-25T19:29:40.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.941+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b28004eb0 con 0x7f1b441430c0 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.941+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1b3c001bd0 con 0x7f1b441430c0 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.941+0000 7f1b3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.941+0000 7f1b3affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.941+0000 7f1b3affd640 10 monclient(hunting): dump: 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:40.942 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:40.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.944+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3571148706 0 0) 0x7f1b440abde0 con 0x7f1b4413f710 2024-03-25T19:29:40.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.944+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b44139250 con 0x7f1b4413f710 2024-03-25T19:29:40.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.944+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1b34001ae0 con 0x7f1b4413f710 2024-03-25T19:29:40.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.944+0000 7f1b3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.944+0000 7f1b3affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.944+0000 7f1b3affd640 10 monclient(hunting): dump: 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:40.945 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:40.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 400411787 0 0) 0x7f1b440ac0c0 con 0x7f1b440a98f0 2024-03-25T19:29:40.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 >> v1:172.21.15.179:6789/0 conn(0x7f1b4413f710 legacy=0x7f1b440ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 >> v1:172.21.15.52:6789/0 conn(0x7f1b441430c0 legacy=0x7f1b441e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:40.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b3affd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:40.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b3affd640 1 monclient: found mon.c 2024-03-25T19:29:40.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:40.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b3affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:40.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b440acbf0 con 0x7f1b440a98f0 2024-03-25T19:29:40.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b3affd640 10 monclient: _finish_auth 0 2024-03-25T19:29:40.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b3affd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:40.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:40.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b4c7c6640 5 monclient: authenticate success, global_id 4316 2024-03-25T19:29:40.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b4c7c6640 10 monclient: _renew_subs 2024-03-25T19:29:40.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b4c7c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:40.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/1044209363 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b44137de0 con 0x7f1b440a98f0 2024-03-25T19:29:40.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b4c7c6640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:40.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b4c7c6640 10 monclient: _renew_subs 2024-03-25T19:29:40.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b4c7c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:40.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/1044209363 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b441368c0 con 0x7f1b440a98f0 2024-03-25T19:29:40.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.953+0000 7f1b4c7c6640 1 librados: init done 2024-03-25T19:29:40.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.954+0000 7f1b4c7c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:40.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.954+0000 7f1b4c7c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:40.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.954+0000 7f1b4c7c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:40.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:40.954+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/1044209363 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b44136b70 con 0x7f1b440a98f0 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.270+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b40003610 con 0x7f1b440a98f0 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.270+0000 7f1b3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.270+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1b400039e0 con 0x7f1b440a98f0 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.270+0000 7f1b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.270+0000 7f1b3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.270+0000 7f1b3affd640 10 monclient: dump: 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:41.271 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:41.272 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:41.272 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:41.272 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:41.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.270+0000 7f1b48d5f640 4 set_mon_vals no callback set 2024-03-25T19:29:41.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.271+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f1b40003c00 con 0x7f1b440a98f0 2024-03-25T19:29:41.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.272+0000 7f1b49d61640 10 monclient: get_auth_request con 0x7f1b2806e0f0 auth_method 0 2024-03-25T19:29:41.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.272+0000 7f1b3affd640 10 client.4316.objecter ms_handle_connect 0x7f1b2806e0f0 2024-03-25T19:29:41.279 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.279+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f1b40054760 con 0x7f1b440a98f0 2024-03-25T19:29:41.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.279+0000 7f1b3affd640 10 client.4316.objecter ms_dispatch 0x7f1b44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:41.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.279+0000 7f1b3affd640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:41.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.279+0000 7f1b3affd640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:41.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.279+0000 7f1b3affd640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-25T19:29:41.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.279+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b40050270 con 0x7f1b440a98f0 2024-03-25T19:29:41.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.279+0000 7f1b3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:41.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.279+0000 7f1b3affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:41.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.526+0000 7f1b4c7c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:41.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.526+0000 7f1b4c7c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:41.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.526+0000 7f1b4c7c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:29:41.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.526+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/1044209363 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b4414f2d0 con 0x7f1b440a98f0 2024-03-25T19:29:41.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.565+0000 7f1b3affd640 1 -- 172.21.15.52:0/1044209363 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f1b400521a0 con 0x7f1b440a98f0 2024-03-25T19:29:41.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.565+0000 7f1b3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:41.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.565+0000 7f1b3affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:41.566 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:29:41.567 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:41.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 10 librados: watch_flush enter 2024-03-25T19:29:41.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 10 librados: watch_flush exit 2024-03-25T19:29:41.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-25T19:29:41.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 10 client.4316.objecter successfully canceled tick 2024-03-25T19:29:41.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/1044209363 >> v1:172.21.15.52:6800/3967427147 conn(0x7f1b2806e0f0 legacy=0x7f1b280705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 10 monclient: shutdown 2024-03-25T19:29:41.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:41.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/1044209363 >> v1:172.21.15.52:6790/0 conn(0x7f1b440a98f0 legacy=0x7f1b44138980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:41.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/1044209363 shutdown_connections 2024-03-25T19:29:41.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/1044209363 >> 172.21.15.52:0/1044209363 conn(0x7f1b440b7ea0 msgr2=0x7f1b44074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:41.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/1044209363 shutdown_connections 2024-03-25T19:29:41.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 1 -- 172.21.15.52:0/1044209363 wait complete. 2024-03-25T19:29:41.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:41.568+0000 7f1b4c7c6640 1 librados: shutdown 2024-03-25T19:29:41.580 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:46.138 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:02:39.737 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:46.142 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:02:39.741 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:46.183 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:02:39.782 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:46.186 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:02:39.728 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:46.196 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:02:39.739 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:46.199 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:02:39.798 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:46.209 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:02:39.808 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:46.225 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:02:39.767 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:46.250 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:02:39.849 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:48.387 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:48.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.497+0000 7fd398a14640 10 monclient: get_monmap_and_config 2024-03-25T19:29:48.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.498+0000 7fd398a14640 10 monclient: build_initial_monmap 2024-03-25T19:29:48.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.498+0000 7fd398a14640 10 monclient: monmap: 2024-03-25T19:29:48.499 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:48.499 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:48.499 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:29:48.499578+0000 2024-03-25T19:29:48.499 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:29:48.499578+0000 2024-03-25T19:29:48.499 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:48.499 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:48.499 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:48.499 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:29:48.499 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:29:48.499 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:48.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.498+0000 7fd398a14640 1 Processor -- start 2024-03-25T19:29:48.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.498+0000 7fd398a14640 1 -- start start 2024-03-25T19:29:48.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.498+0000 7fd398a14640 10 monclient: init 2024-03-25T19:29:48.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.499+0000 7fd398a14640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:48.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.499+0000 7fd398a14640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:48.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.499+0000 7fd398a14640 10 monclient(hunting): picked mon.noname-c con 0x7fd3940b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:48.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.499+0000 7fd398a14640 10 monclient(hunting): picked mon.noname-a con 0x7fd3940b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:48.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.499+0000 7fd398a14640 10 monclient(hunting): picked mon.noname-b con 0x7fd39414be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:48.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.499+0000 7fd398a14640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3940af940 con 0x7fd3940b1730 2024-03-25T19:29:48.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.499+0000 7fd398a14640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3940b0a70 con 0x7fd3940b4d60 2024-03-25T19:29:48.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.499+0000 7fd398a14640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3940b0d50 con 0x7fd39414be90 2024-03-25T19:29:48.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.499+0000 7fd398a14640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:48.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.499+0000 7fd398a14640 10 monclient(hunting): authenticate will time out at 6036.680845s 2024-03-25T19:29:48.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.514+0000 7fd39259c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd3940b4d60 0x7fd39413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:44866/0 (socket says 172.21.15.52:44866) 2024-03-25T19:29:48.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.514+0000 7fd39259c640 1 -- 172.21.15.52:0/1055574453 learned_addr learned my addr 172.21.15.52:0/1055574453 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:48.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.534+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669724161 0 0) 0x7fd3940b0d50 con 0x7fd39414be90 2024-03-25T19:29:48.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.534+0000 7fd39159a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:48.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.534+0000 7fd39159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:48.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.534+0000 7fd39159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:48.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.534+0000 7fd39159a640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:29:48.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.534+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd37c003520 con 0x7fd39414be90 2024-03-25T19:29:48.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.546+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033625178 0 0) 0x7fd3940af940 con 0x7fd3940b1730 2024-03-25T19:29:48.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.546+0000 7fd39159a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:48.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.546+0000 7fd39159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:48.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.546+0000 7fd39159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:48.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.546+0000 7fd39159a640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:29:48.546 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.546+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3940b0d50 con 0x7fd3940b1730 2024-03-25T19:29:48.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.550+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272919815 0 0) 0x7fd3940b0a70 con 0x7fd3940b4d60 2024-03-25T19:29:48.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.550+0000 7fd39159a640 10 monclient(hunting): _negotiate 2024-03-25T19:29:48.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.550+0000 7fd39159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:48.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.550+0000 7fd39159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:48.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.550+0000 7fd39159a640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:29:48.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.550+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3940af940 con 0x7fd3940b4d60 2024-03-25T19:29:48.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3516010518 0 0) 0x7fd37c003520 con 0x7fd39414be90 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 >> v1:172.21.15.179:6789/0 conn(0x7fd3940b4d60 legacy=0x7fd39413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 >> v1:172.21.15.52:6789/0 conn(0x7fd3940b1730 legacy=0x7fd3940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 10 monclient: _finish_hunting 0 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 1 monclient: found mon.noname-b 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3940b1030 con 0x7fd39414be90 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 10 monclient: _finish_auth 0 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 10 monclient: _check_auth_tickets 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd38c001a10 con 0x7fd39414be90 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd398a14640 5 monclient: authenticate success, global_id 4319 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd398a14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:48.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd39159a640 10 monclient: dump: 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:48.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.552+0000 7fd398a14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:48.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.576+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd38c001b70 con 0x7fd39414be90 2024-03-25T19:29:48.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.576+0000 7fd39159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:48.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.576+0000 7fd39159a640 1 -- 172.21.15.52:0/1055574453 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd38c001f80 con 0x7fd39414be90 2024-03-25T19:29:48.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.576+0000 7fd398a14640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:48.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.576+0000 7fd398a14640 4 set_mon_vals no callback set 2024-03-25T19:29:48.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.576+0000 7fd39159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:48.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.576+0000 7fd39159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:48.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.576+0000 7fd39159a640 10 monclient: dump: 2024-03-25T19:29:48.577 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:48.578 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:48.578 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:48.578 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:48.578 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:48.578 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:48.578 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:48.578 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:48.578 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:48.578 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:48.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.577+0000 7fd398a14640 10 monclient: shutdown 2024-03-25T19:29:48.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.577+0000 7fd398a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.577+0000 7fd398a14640 1 -- 172.21.15.52:0/1055574453 >> v1:172.21.15.52:6790/0 conn(0x7fd39414be90 legacy=0x7fd39414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.577+0000 7fd398a14640 1 -- 172.21.15.52:0/1055574453 shutdown_connections 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.577+0000 7fd398a14640 1 -- 172.21.15.52:0/1055574453 >> 172.21.15.52:0/1055574453 conn(0x7fd3940617a0 msgr2=0x7fd394061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.578+0000 7fd398a14640 1 -- 172.21.15.52:0/1055574453 shutdown_connections 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.578+0000 7fd398a14640 1 -- 172.21.15.52:0/1055574453 wait complete. 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.578+0000 7fd398a14640 10 monclient: build_initial_monmap 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.578+0000 7fd398a14640 10 monclient: monmap: 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:48.581 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 1 librados: starting msgr at 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 1 librados: starting objecter 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 1 Processor -- start 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 1 -- start start 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 1 librados: setting wanted keys 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 1 librados: calling monclient init 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 10 monclient: init 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 10 monclient(hunting): picked mon.noname-b con 0x7fd3940b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 10 monclient(hunting): picked mon.noname-a con 0x7fd3940b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.579+0000 7fd398a14640 10 monclient(hunting): picked mon.noname-c con 0x7fd39414be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:48.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.580+0000 7fd398a14640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd394140760 con 0x7fd3940b4d60 2024-03-25T19:29:48.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.580+0000 7fd398a14640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd394140380 con 0x7fd3940b1730 2024-03-25T19:29:48.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.580+0000 7fd398a14640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd394140510 con 0x7fd39414be90 2024-03-25T19:29:48.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.580+0000 7fd398a14640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:48.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.580+0000 7fd398a14640 10 monclient(hunting): authenticate will time out at 6036.760956s 2024-03-25T19:29:48.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.592+0000 7fd392d9d640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fd39414be90 0x7fd3941e20d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:33524/0 (socket says 172.21.15.52:33524) 2024-03-25T19:29:48.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.592+0000 7fd392d9d640 1 -- 172.21.15.52:0/174883452 learned_addr learned my addr 172.21.15.52:0/174883452 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:48.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.595+0000 7fd382ffd640 10 client.?.objecter ms_handle_connect 0x7fd3940b4d60 2024-03-25T19:29:48.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.595+0000 7fd382ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:48.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.597+0000 7fd382ffd640 10 client.?.objecter ms_handle_connect 0x7fd39414be90 2024-03-25T19:29:48.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.597+0000 7fd382ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:48.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.602+0000 7fd382ffd640 10 client.?.objecter ms_handle_connect 0x7fd3940b1730 2024-03-25T19:29:48.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.602+0000 7fd382ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:48.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.611+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301336187 0 0) 0x7fd394140380 con 0x7fd3940b1730 2024-03-25T19:29:48.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.611+0000 7fd382ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:48.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.611+0000 7fd382ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:48.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.611+0000 7fd382ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:48.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.611+0000 7fd382ffd640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:29:48.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.611+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd374004eb0 con 0x7fd3940b1730 2024-03-25T19:29:48.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.611+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156663514 0 0) 0x7fd394140510 con 0x7fd39414be90 2024-03-25T19:29:48.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.611+0000 7fd382ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:48.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.611+0000 7fd382ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:48.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.611+0000 7fd382ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:48.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.611+0000 7fd382ffd640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:29:48.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.612+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd394140380 con 0x7fd39414be90 2024-03-25T19:29:48.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.617+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724869398 0 0) 0x7fd394140760 con 0x7fd3940b4d60 2024-03-25T19:29:48.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.617+0000 7fd382ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:48.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.617+0000 7fd382ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:48.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.617+0000 7fd382ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:48.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.617+0000 7fd382ffd640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:29:48.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.617+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd394140510 con 0x7fd3940b4d60 2024-03-25T19:29:48.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.622+0000 7fd391d9b640 1 -- 172.21.15.52:0/174883452 >> v1:172.21.15.52:6789/0 conn(0x7fd3940b4d60 legacy=0x7fd3941db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T19:29:48.622 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.622+0000 7fd391d9b640 1 -- 172.21.15.52:0/174883452 >> v1:172.21.15.52:6789/0 conn(0x7fd3940b4d60 legacy=0x7fd3941db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:29:48.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.622+0000 7fd391d9b640 1 --1- 172.21.15.52:0/174883452 >> v1:172.21.15.52:6789/0 conn(0x7fd3940b4d60 0x7fd3941db2a0 :-1 s=OPENED pgs=13430 cs=1 l=1).handle_message read tag failed 2024-03-25T19:29:48.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.622+0000 7fd391d9b640 1 --1- 172.21.15.52:0/174883452 >> v1:172.21.15.52:6789/0 conn(0x7fd3940b4d60 0x7fd3941db2a0 :-1 s=OPENED pgs=13430 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:29:48.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.622+0000 7fd382ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.52:6789/0 2024-03-25T19:29:48.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.625+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 357463807 0 0) 0x7fd374004eb0 con 0x7fd3940b1730 2024-03-25T19:29:48.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.626+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd394140760 con 0x7fd3940b1730 2024-03-25T19:29:48.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.626+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd3880018f0 con 0x7fd3940b1730 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.626+0000 7fd382ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.626+0000 7fd382ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.626+0000 7fd382ffd640 10 monclient(hunting): dump: 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:48.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.626+0000 7fd382ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:48.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.628+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2621169025 0 0) 0x7fd394140380 con 0x7fd39414be90 2024-03-25T19:29:48.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.628+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd374004eb0 con 0x7fd39414be90 2024-03-25T19:29:48.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.628+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd38c001e80 con 0x7fd39414be90 2024-03-25T19:29:48.628 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.628+0000 7fd382ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.628+0000 7fd382ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.628+0000 7fd382ffd640 10 monclient(hunting): dump: 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:48.629 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:48.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3003935411 0 0) 0x7fd394140760 con 0x7fd3940b1730 2024-03-25T19:29:48.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 >> v1:172.21.15.52:6790/0 conn(0x7fd39414be90 legacy=0x7fd3941e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 >> v1:172.21.15.52:6789/0 conn(0x7fd3940b4d60 legacy=0x7fd3941db2a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd382ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd382ffd640 1 monclient: found mon.b 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd382ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd382ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd394137530 con 0x7fd3940b1730 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd382ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd382ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd382ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd398a14640 5 monclient: authenticate success, global_id 4333 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd398a14640 10 monclient: _renew_subs 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd398a14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.643+0000 7fd398a14640 1 -- 172.21.15.52:0/174883452 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd394138070 con 0x7fd3940b1730 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.644+0000 7fd398a14640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:48.645 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.644+0000 7fd398a14640 10 monclient: _renew_subs 2024-03-25T19:29:48.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.644+0000 7fd398a14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:48.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.644+0000 7fd398a14640 1 -- 172.21.15.52:0/174883452 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd394145000 con 0x7fd3940b1730 2024-03-25T19:29:48.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.644+0000 7fd398a14640 1 librados: init done 2024-03-25T19:29:48.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.645+0000 7fd398a14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:48.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.645+0000 7fd398a14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:48.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.645+0000 7fd398a14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:48.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.645+0000 7fd398a14640 1 -- 172.21.15.52:0/174883452 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3941408f0 con 0x7fd3940b1730 2024-03-25T19:29:48.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.953+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd388001d40 con 0x7fd3940b1730 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.953+0000 7fd382ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.953+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd388004a00 con 0x7fd3940b1730 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.953+0000 7fd382ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.953+0000 7fd382ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.953+0000 7fd382ffd640 10 monclient: dump: 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:48.954 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:48.955 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:48.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.953+0000 7fd390d99640 4 set_mon_vals no callback set 2024-03-25T19:29:48.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.954+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fd38801aa90 con 0x7fd3940b1730 2024-03-25T19:29:48.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.954+0000 7fd391d9b640 10 monclient: get_auth_request con 0x7fd37406df60 auth_method 0 2024-03-25T19:29:48.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.955+0000 7fd382ffd640 10 client.4333.objecter ms_handle_connect 0x7fd37406df60 2024-03-25T19:29:48.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.959+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fd388054780 con 0x7fd3940b1730 2024-03-25T19:29:48.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.959+0000 7fd382ffd640 10 client.4333.objecter ms_dispatch 0x7fd394000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:48.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.959+0000 7fd382ffd640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:48.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.959+0000 7fd382ffd640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:48.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.959+0000 7fd382ffd640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-25T19:29:48.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.960+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3880500c0 con 0x7fd3940b1730 2024-03-25T19:29:48.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.960+0000 7fd382ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:48.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:48.960+0000 7fd382ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:49.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.215+0000 7fd398a14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:49.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.215+0000 7fd398a14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:49.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.215+0000 7fd398a14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:49.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.215+0000 7fd398a14640 1 -- 172.21.15.52:0/174883452 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3940627b0 con 0x7fd3940b1730 2024-03-25T19:29:49.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.331+0000 7fd382ffd640 1 -- 172.21.15.52:0/174883452 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fd3880521c0 con 0x7fd3940b1730 2024-03-25T19:29:49.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.331+0000 7fd382ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:49.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.331+0000 7fd382ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:49.332 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:29:49.333 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:49.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.333+0000 7fd398a14640 10 librados: watch_flush enter 2024-03-25T19:29:49.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.333+0000 7fd398a14640 10 librados: watch_flush exit 2024-03-25T19:29:49.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.333+0000 7fd398a14640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-25T19:29:49.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.333+0000 7fd398a14640 10 client.4333.objecter successfully canceled tick 2024-03-25T19:29:49.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.333+0000 7fd398a14640 1 -- 172.21.15.52:0/174883452 >> v1:172.21.15.52:6800/3967427147 conn(0x7fd37406df60 legacy=0x7fd374070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:49.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.333+0000 7fd398a14640 10 monclient: shutdown 2024-03-25T19:29:49.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.333+0000 7fd398a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:49.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.333+0000 7fd398a14640 1 -- 172.21.15.52:0/174883452 >> v1:172.21.15.179:6789/0 conn(0x7fd3940b1730 legacy=0x7fd3941dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:49.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.333+0000 7fd398a14640 1 -- 172.21.15.52:0/174883452 shutdown_connections 2024-03-25T19:29:49.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.333+0000 7fd398a14640 1 -- 172.21.15.52:0/174883452 >> 172.21.15.52:0/174883452 conn(0x7fd3940617a0 msgr2=0x7fd39413c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:49.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.334+0000 7fd398a14640 1 -- 172.21.15.52:0/174883452 shutdown_connections 2024-03-25T19:29:49.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.334+0000 7fd398a14640 1 -- 172.21.15.52:0/174883452 wait complete. 2024-03-25T19:29:49.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:49.334+0000 7fd398a14640 1 librados: shutdown 2024-03-25T19:29:49.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:56.152 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:56.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.270+0000 7f04069e2640 10 monclient: get_monmap_and_config 2024-03-25T19:29:56.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.270+0000 7f04069e2640 10 monclient: build_initial_monmap 2024-03-25T19:29:56.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.270+0000 7f04069e2640 10 monclient: monmap: 2024-03-25T19:29:56.358 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:56.358 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:56.358 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:29:56.272250+0000 2024-03-25T19:29:56.358 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:29:56.272250+0000 2024-03-25T19:29:56.358 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:56.358 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:56.358 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:56.358 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:29:56.358 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:29:56.358 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:56.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.271+0000 7f04069e2640 1 Processor -- start 2024-03-25T19:29:56.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.271+0000 7f04069e2640 1 -- start start 2024-03-25T19:29:56.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.271+0000 7f04069e2640 10 monclient: init 2024-03-25T19:29:56.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.272+0000 7f04069e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:56.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.272+0000 7f04069e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.272+0000 7f04069e2640 10 monclient(hunting): picked mon.noname-c con 0x7f04000a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.272+0000 7f04069e2640 10 monclient(hunting): picked mon.noname-b con 0x7f040013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.272+0000 7f04069e2640 10 monclient(hunting): picked mon.noname-a con 0x7f04001430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.272+0000 7f04069e2640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0400142520 con 0x7f04001430c0 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.272+0000 7f04069e2640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04000b28a0 con 0x7f04000a98f0 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.272+0000 7f04069e2640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04000b2a30 con 0x7f040013f710 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.272+0000 7f04069e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.272+0000 7f04069e2640 10 monclient(hunting): authenticate will time out at 6044.453363s 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.279+0000 7f03fffff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f04000a98f0 0x7f04000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:48084/0 (socket says 172.21.15.52:48084) 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.279+0000 7f03fffff640 1 -- 172.21.15.52:0/715092447 learned_addr learned my addr 172.21.15.52:0/715092447 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.296+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358155184 0 0) 0x7f04000b28a0 con 0x7f04000a98f0 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.296+0000 7f03feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.296+0000 7f03feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.296+0000 7f03feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.296+0000 7f03feffd640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.296+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e8003520 con 0x7f04000a98f0 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.306+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745798657 0 0) 0x7f04000b2a30 con 0x7f040013f710 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.306+0000 7f03feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.306+0000 7f03feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.306+0000 7f03feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.306+0000 7f03feffd640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.306+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04000b28a0 con 0x7f040013f710 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.309+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169652164 0 0) 0x7f0400142520 con 0x7f04001430c0 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.309+0000 7f03feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.309+0000 7f03feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.309+0000 7f03feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.309+0000 7f03feffd640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.309+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04000b2a30 con 0x7f04001430c0 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2609948560 0 0) 0x7f03e8003520 con 0x7f04000a98f0 2024-03-25T19:29:56.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 >> v1:172.21.15.52:6790/0 conn(0x7f040013f710 legacy=0x7f04001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 >> v1:172.21.15.52:6789/0 conn(0x7f04001430c0 legacy=0x7f040014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 1 monclient: found mon.noname-c 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04000b44c0 con 0x7f04000a98f0 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f03f0001690 con 0x7f04000a98f0 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f04069e2640 5 monclient: authenticate success, global_id 4336 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f04069e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f03feffd640 10 monclient: dump: 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.311+0000 7f04069e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.333+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03f00017f0 con 0x7f04000a98f0 2024-03-25T19:29:56.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.333+0000 7f03feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.333+0000 7f03feffd640 1 -- 172.21.15.52:0/715092447 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f03f0001c00 con 0x7f04000a98f0 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.333+0000 7f04069e2640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.333+0000 7f04069e2640 4 set_mon_vals no callback set 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.333+0000 7f03feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.333+0000 7f03feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.333+0000 7f03feffd640 10 monclient: dump: 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:56.368 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.334+0000 7f04069e2640 10 monclient: shutdown 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.334+0000 7f04069e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.334+0000 7f04069e2640 1 -- 172.21.15.52:0/715092447 >> v1:172.21.15.179:6789/0 conn(0x7f04000a98f0 legacy=0x7f04000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.334+0000 7f04069e2640 1 -- 172.21.15.52:0/715092447 shutdown_connections 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.334+0000 7f04069e2640 1 -- 172.21.15.52:0/715092447 >> 172.21.15.52:0/715092447 conn(0x7f04000b7ea0 msgr2=0x7f04000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.334+0000 7f04069e2640 1 -- 172.21.15.52:0/715092447 shutdown_connections 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.334+0000 7f04069e2640 1 -- 172.21.15.52:0/715092447 wait complete. 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.335+0000 7f04069e2640 10 monclient: build_initial_monmap 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.335+0000 7f04069e2640 10 monclient: monmap: 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:29:56.369 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.335+0000 7f04069e2640 1 librados: starting msgr at 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.335+0000 7f04069e2640 1 librados: starting objecter 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.335+0000 7f04069e2640 1 Processor -- start 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 1 -- start start 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 1 librados: setting wanted keys 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 1 librados: calling monclient init 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 10 monclient: init 2024-03-25T19:29:56.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 10 monclient(hunting): picked mon.noname-a con 0x7f04000a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 10 monclient(hunting): picked mon.noname-c con 0x7f040013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 10 monclient(hunting): picked mon.noname-b con 0x7f04001430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04001daba0 con 0x7f04000a98f0 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04000abac0 con 0x7f04001430c0 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04001db740 con 0x7f040013f710 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.336+0000 7f04069e2640 10 monclient(hunting): authenticate will time out at 6044.517678s 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.347+0000 7f0404f7f640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f04001430c0 0x7f04001e1f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:48126/0 (socket says 172.21.15.52:48126) 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.347+0000 7f0404f7f640 1 -- 172.21.15.52:0/2129875437 learned_addr learned my addr 172.21.15.52:0/2129875437 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.350+0000 7f03fcff9640 10 client.?.objecter ms_handle_connect 0x7f04001430c0 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.350+0000 7f03fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.352+0000 7f03fcff9640 10 client.?.objecter ms_handle_connect 0x7f040013f710 2024-03-25T19:29:56.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.352+0000 7f03fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.355+0000 7f03fcff9640 10 client.?.objecter ms_handle_connect 0x7f04000a98f0 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.355+0000 7f03fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.361+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068692657 0 0) 0x7f04000abac0 con 0x7f04001430c0 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.361+0000 7f03fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.361+0000 7f03fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.361+0000 7f03fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.361+0000 7f03fcff9640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.361+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03ec004eb0 con 0x7f04001430c0 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.362+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847580109 0 0) 0x7f04001db740 con 0x7f040013f710 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.362+0000 7f03fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.362+0000 7f03fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.362+0000 7f03fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.362+0000 7f03fcff9640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:29:56.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.362+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04000abac0 con 0x7f040013f710 2024-03-25T19:29:56.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.368+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328573621 0 0) 0x7f04001daba0 con 0x7f04000a98f0 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.368+0000 7f03fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.368+0000 7f03fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.368+0000 7f03fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.368+0000 7f03fcff9640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.368+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04001db740 con 0x7f04000a98f0 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.372+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3649536288 0 0) 0x7f03ec004eb0 con 0x7f04001430c0 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.372+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04001daba0 con 0x7f04001430c0 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.372+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f03f8001c70 con 0x7f04001430c0 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.372+0000 7f03fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.372+0000 7f03fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.372+0000 7f03fcff9640 10 monclient(hunting): dump: 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:56.374 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:56.375 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:56.375 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:56.375 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.375 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:56.375 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:56.375 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:56.375 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:56.375 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:56.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.372+0000 7f03fcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4121008374 0 0) 0x7f04001daba0 con 0x7f04001430c0 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 >> v1:172.21.15.52:6790/0 conn(0x7f040013f710 legacy=0x7f0400138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 >> v1:172.21.15.52:6789/0 conn(0x7f04000a98f0 legacy=0x7f04001db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f03fcff9640 10 monclient: _finish_hunting 0 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f03fcff9640 1 monclient: found mon.b 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f03fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f03fcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04001db8d0 con 0x7f04001430c0 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f03fcff9640 10 monclient: _finish_auth 0 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f03fcff9640 10 monclient: _check_auth_tickets 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f03fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f04069e2640 5 monclient: authenticate success, global_id 4339 2024-03-25T19:29:56.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f04069e2640 10 monclient: _renew_subs 2024-03-25T19:29:56.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f04069e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:56.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f04069e2640 1 -- 172.21.15.52:0/2129875437 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0400138c10 con 0x7f04001430c0 2024-03-25T19:29:56.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f04069e2640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:56.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f04069e2640 10 monclient: _renew_subs 2024-03-25T19:29:56.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f04069e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:56.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f04069e2640 1 -- 172.21.15.52:0/2129875437 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0400135130 con 0x7f04001430c0 2024-03-25T19:29:56.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.384+0000 7f04069e2640 1 librados: init done 2024-03-25T19:29:56.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.385+0000 7f04069e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:56.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.385+0000 7f04069e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:56.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.385+0000 7f04069e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:56.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.385+0000 7f04069e2640 1 -- 172.21.15.52:0/2129875437 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04000627b0 con 0x7f04001430c0 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.672+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03f8003970 con 0x7f04001430c0 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.672+0000 7f03fcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.672+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f03f8003d80 con 0x7f04001430c0 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.672+0000 7f03fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.672+0000 7f03fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.672+0000 7f03fcff9640 10 monclient: dump: 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.673 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:29:56.674 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:29:56.674 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:29:56.674 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:29:56.674 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:29:56.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.672+0000 7f03fe7fc640 4 set_mon_vals no callback set 2024-03-25T19:29:56.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.673+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f03f8003fa0 con 0x7f04001430c0 2024-03-25T19:29:56.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.674+0000 7f03fffff640 10 monclient: get_auth_request con 0x7f03ec06ddf0 auth_method 0 2024-03-25T19:29:56.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.674+0000 7f03fcff9640 10 client.4339.objecter ms_handle_connect 0x7f03ec06ddf0 2024-03-25T19:29:56.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.682+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f03f8054ac0 con 0x7f04001430c0 2024-03-25T19:29:56.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.682+0000 7f03fcff9640 10 client.4339.objecter ms_dispatch 0x7f0400000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:56.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.682+0000 7f03fcff9640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:56.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.682+0000 7f03fcff9640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:56.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.682+0000 7f03fcff9640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-25T19:29:56.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.694+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03f80505d0 con 0x7f04001430c0 2024-03-25T19:29:56.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.694+0000 7f03fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:56.695 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.694+0000 7f03fcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:56.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.917+0000 7f04069e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:56.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.917+0000 7f04069e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:56.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.917+0000 7f04069e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:29:56.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:56.917+0000 7f04069e2640 1 -- 172.21.15.52:0/2129875437 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0400138da0 con 0x7f04001430c0 2024-03-25T19:29:57.039 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.038+0000 7f03fcff9640 1 -- 172.21.15.52:0/2129875437 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f03f8052500 con 0x7f04001430c0 2024-03-25T19:29:57.039 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.038+0000 7f03fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:57.039 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.038+0000 7f03fcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:57.039 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:29:57.040 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:57.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 10 librados: watch_flush enter 2024-03-25T19:29:57.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 10 librados: watch_flush exit 2024-03-25T19:29:57.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-25T19:29:57.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 10 client.4339.objecter successfully canceled tick 2024-03-25T19:29:57.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 1 -- 172.21.15.52:0/2129875437 >> v1:172.21.15.52:6800/3967427147 conn(0x7f03ec06ddf0 legacy=0x7f03ec0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:57.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 10 monclient: shutdown 2024-03-25T19:29:57.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:57.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 1 -- 172.21.15.52:0/2129875437 >> v1:172.21.15.179:6789/0 conn(0x7f04001430c0 legacy=0x7f04001e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:57.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 1 -- 172.21.15.52:0/2129875437 shutdown_connections 2024-03-25T19:29:57.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 1 -- 172.21.15.52:0/2129875437 >> 172.21.15.52:0/2129875437 conn(0x7f04000b7ea0 msgr2=0x7f0400074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:57.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 1 -- 172.21.15.52:0/2129875437 shutdown_connections 2024-03-25T19:29:57.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 1 -- 172.21.15.52:0/2129875437 wait complete. 2024-03-25T19:29:57.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:29:57.041+0000 7f04069e2640 1 librados: shutdown 2024-03-25T19:29:57.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:03.859 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:03.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.977+0000 7f1bcd27b640 10 monclient: get_monmap_and_config 2024-03-25T19:30:03.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.977+0000 7f1bcd27b640 10 monclient: build_initial_monmap 2024-03-25T19:30:03.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.977+0000 7f1bcd27b640 10 monclient: monmap: 2024-03-25T19:30:03.976 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:03.976 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:03.976 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:30:03.977605+0000 2024-03-25T19:30:03.976 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:30:03.977605+0000 2024-03-25T19:30:03.977 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:03.977 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:03.977 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:03.977 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:30:03.977 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:30:03.977 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:03.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.977+0000 7f1bcd27b640 1 Processor -- start 2024-03-25T19:30:03.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 1 -- start start 2024-03-25T19:30:03.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 10 monclient: init 2024-03-25T19:30:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 10 monclient(hunting): picked mon.noname-a con 0x7f1bc80a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc813f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc81430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc8142520 con 0x7f1bc80a98f0 2024-03-25T19:30:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc80b28a0 con 0x7f1bc813f710 2024-03-25T19:30:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc80b2a30 con 0x7f1bc81430c0 2024-03-25T19:30:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:03.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.978+0000 7f1bcd27b640 10 monclient(hunting): authenticate will time out at 6052.158906s 2024-03-25T19:30:03.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.995+0000 7f1bc77fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f1bc81430c0 0x7f1bc814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:51994/0 (socket says 172.21.15.52:51994) 2024-03-25T19:30:03.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:03.995+0000 7f1bc77fe640 1 -- 172.21.15.52:0/1152829436 learned_addr learned my addr 172.21.15.52:0/1152829436 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:04.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.014+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428765001 0 0) 0x7f1bc80b2a30 con 0x7f1bc81430c0 2024-03-25T19:30:04.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.014+0000 7f1bc67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.014+0000 7f1bc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.014+0000 7f1bc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.014+0000 7f1bc67fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:30:04.014 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.014+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb4003520 con 0x7f1bc81430c0 2024-03-25T19:30:04.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.017+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783772938 0 0) 0x7f1bc80b28a0 con 0x7f1bc813f710 2024-03-25T19:30:04.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.017+0000 7f1bc67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.017+0000 7f1bc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.017+0000 7f1bc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.017+0000 7f1bc67fc640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:30:04.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.017+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc80b2a30 con 0x7f1bc813f710 2024-03-25T19:30:04.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.020+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517070342 0 0) 0x7f1bc8142520 con 0x7f1bc80a98f0 2024-03-25T19:30:04.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.020+0000 7f1bc67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.020+0000 7f1bc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.020+0000 7f1bc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.020+0000 7f1bc67fc640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:30:04.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.020+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc80b28a0 con 0x7f1bc80a98f0 2024-03-25T19:30:04.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3376386623 0 0) 0x7f1bc80b28a0 con 0x7f1bc80a98f0 2024-03-25T19:30:04.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 >> v1:172.21.15.52:6790/0 conn(0x7f1bc81430c0 legacy=0x7f1bc814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 >> v1:172.21.15.179:6789/0 conn(0x7f1bc813f710 legacy=0x7f1bc81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:04.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 1 monclient: found mon.noname-a 2024-03-25T19:30:04.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:04.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:30:04.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc80b44c0 con 0x7f1bc80a98f0 2024-03-25T19:30:04.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1bb80015b0 con 0x7f1bc80a98f0 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bcd27b640 5 monclient: authenticate success, global_id 4332 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bcd27b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bc67fc640 10 monclient: dump: 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:04.027 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:04.027 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:04.027 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:04.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.025+0000 7f1bcd27b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.031+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bb8001710 con 0x7f1bc80a98f0 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.031+0000 7f1bc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.031+0000 7f1bc67fc640 1 -- 172.21.15.52:0/1152829436 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1bb8001b60 con 0x7f1bc80a98f0 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.031+0000 7f1bc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.031+0000 7f1bc67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.031+0000 7f1bc67fc640 10 monclient: dump: 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:04.032 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:04.032 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:04.032 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:04.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.031+0000 7f1bcd27b640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:04.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.031+0000 7f1bcd27b640 4 set_mon_vals no callback set 2024-03-25T19:30:04.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.032+0000 7f1bcd27b640 10 monclient: shutdown 2024-03-25T19:30:04.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.032+0000 7f1bcd27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:04.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.032+0000 7f1bcd27b640 1 -- 172.21.15.52:0/1152829436 >> v1:172.21.15.52:6789/0 conn(0x7f1bc80a98f0 legacy=0x7f1bc80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.032+0000 7f1bcd27b640 1 -- 172.21.15.52:0/1152829436 shutdown_connections 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.032+0000 7f1bcd27b640 1 -- 172.21.15.52:0/1152829436 >> 172.21.15.52:0/1152829436 conn(0x7f1bc80b7ea0 msgr2=0x7f1bc80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.032+0000 7f1bcd27b640 1 -- 172.21.15.52:0/1152829436 shutdown_connections 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.032+0000 7f1bcd27b640 1 -- 172.21.15.52:0/1152829436 wait complete. 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.033+0000 7f1bcd27b640 10 monclient: build_initial_monmap 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.033+0000 7f1bcd27b640 10 monclient: monmap: 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.033+0000 7f1bcd27b640 1 librados: starting msgr at 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.033+0000 7f1bcd27b640 1 librados: starting objecter 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.034+0000 7f1bcd27b640 1 Processor -- start 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.034+0000 7f1bcd27b640 1 -- start start 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.034+0000 7f1bcd27b640 1 librados: setting wanted keys 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.034+0000 7f1bcd27b640 1 librados: calling monclient init 2024-03-25T19:30:04.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.034+0000 7f1bcd27b640 10 monclient: init 2024-03-25T19:30:04.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.035+0000 7f1bcd27b640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:04.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.035+0000 7f1bcd27b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:04.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.035+0000 7f1bcd27b640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc80a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:04.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.035+0000 7f1bcd27b640 10 monclient(hunting): picked mon.noname-a con 0x7f1bc813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:04.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.035+0000 7f1bcd27b640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc81430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:04.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.035+0000 7f1bcd27b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc8136830 con 0x7f1bc813f710 2024-03-25T19:30:04.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.035+0000 7f1bcd27b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc8137e40 con 0x7f1bc81430c0 2024-03-25T19:30:04.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.035+0000 7f1bcd27b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc8138120 con 0x7f1bc80a98f0 2024-03-25T19:30:04.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.035+0000 7f1bcd27b640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:04.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.035+0000 7f1bcd27b640 10 monclient(hunting): authenticate will time out at 6052.215037s 2024-03-25T19:30:04.041 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.041+0000 7f1bc6ffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f1bc80a98f0 0x7f1bc81db0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:52046/0 (socket says 172.21.15.52:52046) 2024-03-25T19:30:04.041 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.041+0000 7f1bc6ffd640 1 -- 172.21.15.52:0/3964735702 learned_addr learned my addr 172.21.15.52:0/3964735702 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:04.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.043+0000 7f1bbf7fe640 10 client.?.objecter ms_handle_connect 0x7f1bc80a98f0 2024-03-25T19:30:04.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.043+0000 7f1bbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:04.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.044+0000 7f1bbf7fe640 10 client.?.objecter ms_handle_connect 0x7f1bc81430c0 2024-03-25T19:30:04.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.044+0000 7f1bbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:04.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.055+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433948619 0 0) 0x7f1bc8138120 con 0x7f1bc80a98f0 2024-03-25T19:30:04.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.055+0000 7f1bbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.055+0000 7f1bbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.055+0000 7f1bbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.055+0000 7f1bbf7fe640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:30:04.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.055+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba8004eb0 con 0x7f1bc80a98f0 2024-03-25T19:30:04.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.057+0000 7f1bbf7fe640 10 client.?.objecter ms_handle_connect 0x7f1bc813f710 2024-03-25T19:30:04.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.057+0000 7f1bbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:04.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.058+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191719963 0 0) 0x7f1bc8137e40 con 0x7f1bc81430c0 2024-03-25T19:30:04.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.058+0000 7f1bbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.058+0000 7f1bbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.058+0000 7f1bbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.058+0000 7f1bbf7fe640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:30:04.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.058+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc8138120 con 0x7f1bc81430c0 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.064+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763845493 0 0) 0x7f1bc8138120 con 0x7f1bc81430c0 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.064+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bc8137e40 con 0x7f1bc81430c0 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.064+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1bc0001ab0 con 0x7f1bc81430c0 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.064+0000 7f1bbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.064+0000 7f1bbf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.064+0000 7f1bbf7fe640 10 monclient(hunting): dump: 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:04.064 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:04.065 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:04.065 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.065 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:04.065 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:04.065 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:04.065 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:04.065 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:04.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.064+0000 7f1bbf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.072+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971566420 0 0) 0x7f1bc8136830 con 0x7f1bc813f710 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.072+0000 7f1bbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.072+0000 7f1bbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.072+0000 7f1bbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:04.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.072+0000 7f1bbf7fe640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.072+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc8138120 con 0x7f1bc813f710 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.072+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1681898125 0 0) 0x7f1ba8004eb0 con 0x7f1bc80a98f0 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.073+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bc8136830 con 0x7f1bc80a98f0 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.073+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1bb8001b20 con 0x7f1bc80a98f0 2024-03-25T19:30:04.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.073+0000 7f1bbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.073+0000 7f1bbf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.073+0000 7f1bbf7fe640 10 monclient(hunting): dump: 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:04.073 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:04.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.078+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3182905221 0 0) 0x7f1bc8137e40 con 0x7f1bc81430c0 2024-03-25T19:30:04.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 >> v1:172.21.15.52:6790/0 conn(0x7f1bc80a98f0 legacy=0x7f1bc81db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 >> v1:172.21.15.52:6789/0 conn(0x7f1bc813f710 legacy=0x7f1bc8138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bbf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:04.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bbf7fe640 1 monclient: found mon.b 2024-03-25T19:30:04.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:04.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bbf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc80abac0 con 0x7f1bc81430c0 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bbf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bbf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bcd27b640 5 monclient: authenticate success, global_id 4345 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bcd27b640 10 monclient: _renew_subs 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bcd27b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bcd27b640 1 -- 172.21.15.52:0/3964735702 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc81db7f0 con 0x7f1bc81430c0 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bcd27b640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bcd27b640 10 monclient: _renew_subs 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bcd27b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bcd27b640 1 -- 172.21.15.52:0/3964735702 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc8137a10 con 0x7f1bc81430c0 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.079+0000 7f1bcd27b640 1 librados: init done 2024-03-25T19:30:04.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.080+0000 7f1bcd27b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:04.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.080+0000 7f1bcd27b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:04.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.080+0000 7f1bcd27b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:04.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.080+0000 7f1bcd27b640 1 -- 172.21.15.52:0/3964735702 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc80627b0 con 0x7f1bc81430c0 2024-03-25T19:30:04.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.101+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bc0001fa0 con 0x7f1bc81430c0 2024-03-25T19:30:04.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.101+0000 7f1bbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:04.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.101+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1bc0004d00 con 0x7f1bc81430c0 2024-03-25T19:30:04.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.101+0000 7f1bbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.101+0000 7f1bbf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:04.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.101+0000 7f1bbf7fe640 10 monclient: dump: 2024-03-25T19:30:04.101 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:04.101 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:04.102 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:04.102 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:04.102 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.102 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:04.102 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:04.102 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:04.102 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:04.102 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:04.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.101+0000 7f1bc5ffb640 4 set_mon_vals no callback set 2024-03-25T19:30:04.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.103+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f1bc001acd0 con 0x7f1bc81430c0 2024-03-25T19:30:04.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.103+0000 7f1bc6ffd640 10 monclient: get_auth_request con 0x7f1ba806df20 auth_method 0 2024-03-25T19:30:04.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.103+0000 7f1bbf7fe640 10 client.4345.objecter ms_handle_connect 0x7f1ba806df20 2024-03-25T19:30:04.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.105+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f1bc0054a00 con 0x7f1bc81430c0 2024-03-25T19:30:04.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.105+0000 7f1bbf7fe640 10 client.4345.objecter ms_dispatch 0x7f1bc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:04.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.105+0000 7f1bbf7fe640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:04.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.105+0000 7f1bbf7fe640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:04.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.105+0000 7f1bbf7fe640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-25T19:30:04.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.353+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bc0050340 con 0x7f1bc81430c0 2024-03-25T19:30:04.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.353+0000 7f1bbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:04.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.353+0000 7f1bbf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:04.604 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.604+0000 7f1bcd27b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:04.604 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.604+0000 7f1bcd27b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:04.604 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.604+0000 7f1bcd27b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:04.604 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.604+0000 7f1bcd27b640 1 -- 172.21.15.52:0/3964735702 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bc8069f40 con 0x7f1bc81430c0 2024-03-25T19:30:04.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.764+0000 7f1bbf7fe640 1 -- 172.21.15.52:0/3964735702 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f1bc0052440 con 0x7f1bc81430c0 2024-03-25T19:30:04.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.764+0000 7f1bbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:04.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.764+0000 7f1bbf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:04.764 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:30:04.765 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:04.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.766+0000 7f1bcd27b640 10 librados: watch_flush enter 2024-03-25T19:30:04.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.766+0000 7f1bcd27b640 10 librados: watch_flush exit 2024-03-25T19:30:04.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.766+0000 7f1bcd27b640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-25T19:30:04.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.766+0000 7f1bcd27b640 10 client.4345.objecter successfully canceled tick 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.766+0000 7f1bcd27b640 1 -- 172.21.15.52:0/3964735702 >> v1:172.21.15.52:6800/3967427147 conn(0x7f1ba806df20 legacy=0x7f1ba80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.766+0000 7f1bcd27b640 10 monclient: shutdown 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.766+0000 7f1bcd27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.766+0000 7f1bcd27b640 1 -- 172.21.15.52:0/3964735702 >> v1:172.21.15.179:6789/0 conn(0x7f1bc81430c0 legacy=0x7f1bc81e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.766+0000 7f1bcd27b640 1 -- 172.21.15.52:0/3964735702 shutdown_connections 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.766+0000 7f1bcd27b640 1 -- 172.21.15.52:0/3964735702 >> 172.21.15.52:0/3964735702 conn(0x7f1bc80b7ea0 msgr2=0x7f1bc813d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.767+0000 7f1bcd27b640 1 -- 172.21.15.52:0/3964735702 shutdown_connections 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.767+0000 7f1bcd27b640 1 -- 172.21.15.52:0/3964735702 wait complete. 2024-03-25T19:30:04.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:04.767+0000 7f1bcd27b640 1 librados: shutdown 2024-03-25T19:30:04.778 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:06.188 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:02:59.787 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:06.196 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:02:59.795 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:06.232 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:02:59.831 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:06.238 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:02:59.780 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:06.249 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:02:59.791 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:06.251 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:02:59.849 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:06.260 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:02:59.859 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:06.269 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:02:59.811 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:06.298 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:02:59.897 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:11.585 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:11.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.704+0000 7f9c846e4640 10 monclient: get_monmap_and_config 2024-03-25T19:30:11.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.704+0000 7f9c846e4640 10 monclient: build_initial_monmap 2024-03-25T19:30:11.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.704+0000 7f9c846e4640 10 monclient: monmap: 2024-03-25T19:30:11.703 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:11.704 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:11.704 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:30:11.704599+0000 2024-03-25T19:30:11.704 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:30:11.704599+0000 2024-03-25T19:30:11.704 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:11.704 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:11.704 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:11.704 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:30:11.704 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:30:11.704 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:11.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.704+0000 7f9c846e4640 1 Processor -- start 2024-03-25T19:30:11.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.704+0000 7f9c846e4640 1 -- start start 2024-03-25T19:30:11.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.704+0000 7f9c846e4640 10 monclient: init 2024-03-25T19:30:11.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.705+0000 7f9c846e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:11.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.705+0000 7f9c846e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:11.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.705+0000 7f9c846e4640 10 monclient(hunting): picked mon.noname-c con 0x7f9c7c0a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:11.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.706+0000 7f9c846e4640 10 monclient(hunting): picked mon.noname-b con 0x7f9c7c13f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:11.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.706+0000 7f9c846e4640 10 monclient(hunting): picked mon.noname-a con 0x7f9c7c1430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:11.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.706+0000 7f9c846e4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7c142520 con 0x7f9c7c1430c0 2024-03-25T19:30:11.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.706+0000 7f9c846e4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7c0b28a0 con 0x7f9c7c0a98f0 2024-03-25T19:30:11.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.706+0000 7f9c846e4640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7c0b2a30 con 0x7f9c7c13f710 2024-03-25T19:30:11.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.706+0000 7f9c846e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:11.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.706+0000 7f9c846e4640 10 monclient(hunting): authenticate will time out at 6059.886089s 2024-03-25T19:30:11.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.709+0000 7f9c81c7f640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f9c7c13f710 0x7f9c7c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:43830/0 (socket says 172.21.15.52:43830) 2024-03-25T19:30:11.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.709+0000 7f9c81c7f640 1 -- 172.21.15.52:0/3687878295 learned_addr learned my addr 172.21.15.52:0/3687878295 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:11.722 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.723+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567651154 0 0) 0x7f9c7c0b28a0 con 0x7f9c7c0a98f0 2024-03-25T19:30:11.722 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.723+0000 7f9c8147e640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.722 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.723+0000 7f9c8147e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.722 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.723+0000 7f9c8147e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.722 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.723+0000 7f9c8147e640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:30:11.722 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.723+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c68003520 con 0x7f9c7c0a98f0 2024-03-25T19:30:11.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.725+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982153790 0 0) 0x7f9c7c0b2a30 con 0x7f9c7c13f710 2024-03-25T19:30:11.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.725+0000 7f9c8147e640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.725+0000 7f9c8147e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.725+0000 7f9c8147e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.725+0000 7f9c8147e640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:30:11.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.725+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c7c0b28a0 con 0x7f9c7c13f710 2024-03-25T19:30:11.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.730+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509551371 0 0) 0x7f9c7c142520 con 0x7f9c7c1430c0 2024-03-25T19:30:11.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.730+0000 7f9c8147e640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.730+0000 7f9c8147e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.730+0000 7f9c8147e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.730+0000 7f9c8147e640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:30:11.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.730+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c7c0b2a30 con 0x7f9c7c1430c0 2024-03-25T19:30:11.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4021440822 0 0) 0x7f9c7c0b28a0 con 0x7f9c7c13f710 2024-03-25T19:30:11.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 >> v1:172.21.15.179:6789/0 conn(0x7f9c7c0a98f0 legacy=0x7f9c7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 >> v1:172.21.15.52:6789/0 conn(0x7f9c7c1430c0 legacy=0x7f9c7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 10 monclient: _finish_hunting 0 2024-03-25T19:30:11.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 1 monclient: found mon.noname-b 2024-03-25T19:30:11.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:11.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c0b44c0 con 0x7f9c7c13f710 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 10 monclient: _finish_auth 0 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 10 monclient: _check_auth_tickets 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9c6c001670 con 0x7f9c7c13f710 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c846e4640 5 monclient: authenticate success, global_id 4337 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c846e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c8147e640 10 monclient: dump: 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:11.735 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:11.736 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.736 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:11.736 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:11.736 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:11.736 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:11.736 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:11.736 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.734+0000 7f9c846e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:11.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.755+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c6c0017d0 con 0x7f9c7c13f710 2024-03-25T19:30:11.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.755+0000 7f9c8147e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:11.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.755+0000 7f9c8147e640 1 -- 172.21.15.52:0/3687878295 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9c6c001be0 con 0x7f9c7c13f710 2024-03-25T19:30:11.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.755+0000 7f9c8147e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.755+0000 7f9c8147e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.755+0000 7f9c8147e640 10 monclient: dump: 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.755+0000 7f9c846e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:11.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.755+0000 7f9c846e4640 4 set_mon_vals no callback set 2024-03-25T19:30:11.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.755+0000 7f9c846e4640 10 monclient: shutdown 2024-03-25T19:30:11.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.755+0000 7f9c846e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:11.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.755+0000 7f9c846e4640 1 -- 172.21.15.52:0/3687878295 >> v1:172.21.15.52:6790/0 conn(0x7f9c7c13f710 legacy=0x7f9c7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.756+0000 7f9c846e4640 1 -- 172.21.15.52:0/3687878295 shutdown_connections 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.756+0000 7f9c846e4640 1 -- 172.21.15.52:0/3687878295 >> 172.21.15.52:0/3687878295 conn(0x7f9c7c0b7ea0 msgr2=0x7f9c7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.756+0000 7f9c846e4640 1 -- 172.21.15.52:0/3687878295 shutdown_connections 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.756+0000 7f9c846e4640 1 -- 172.21.15.52:0/3687878295 wait complete. 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.757+0000 7f9c846e4640 10 monclient: build_initial_monmap 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.757+0000 7f9c846e4640 10 monclient: monmap: 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:30:11.757 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:11.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.757+0000 7f9c846e4640 1 librados: starting msgr at 2024-03-25T19:30:11.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.757+0000 7f9c846e4640 1 librados: starting objecter 2024-03-25T19:30:11.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.757+0000 7f9c846e4640 1 Processor -- start 2024-03-25T19:30:11.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 1 -- start start 2024-03-25T19:30:11.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 1 librados: setting wanted keys 2024-03-25T19:30:11.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 1 librados: calling monclient init 2024-03-25T19:30:11.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 10 monclient: init 2024-03-25T19:30:11.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:11.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:11.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 10 monclient(hunting): picked mon.noname-c con 0x7f9c7c0a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:11.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 10 monclient(hunting): picked mon.noname-b con 0x7f9c7c13f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:11.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 10 monclient(hunting): picked mon.noname-a con 0x7f9c7c1430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:11.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7c0abf10 con 0x7f9c7c1430c0 2024-03-25T19:30:11.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7c137c30 con 0x7f9c7c13f710 2024-03-25T19:30:11.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7c1d7880 con 0x7f9c7c0a98f0 2024-03-25T19:30:11.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:11.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.758+0000 7f9c846e4640 10 monclient(hunting): authenticate will time out at 6059.938910s 2024-03-25T19:30:11.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.773+0000 7f9c82480640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f9c7c0a98f0 0x7f9c7c1d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:43874/0 (socket says 172.21.15.52:43874) 2024-03-25T19:30:11.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.773+0000 7f9c82480640 1 -- 172.21.15.52:0/2667777983 learned_addr learned my addr 172.21.15.52:0/2667777983 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:11.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.775+0000 7f9c72ffd640 10 client.?.objecter ms_handle_connect 0x7f9c7c1430c0 2024-03-25T19:30:11.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.775+0000 7f9c72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:11.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.776+0000 7f9c72ffd640 10 client.?.objecter ms_handle_connect 0x7f9c7c0a98f0 2024-03-25T19:30:11.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.776+0000 7f9c72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:11.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.781+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884844330 0 0) 0x7f9c7c0abf10 con 0x7f9c7c1430c0 2024-03-25T19:30:11.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.782+0000 7f9c72ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.782+0000 7f9c72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.782+0000 7f9c72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.782+0000 7f9c72ffd640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:30:11.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.782+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c60004eb0 con 0x7f9c7c1430c0 2024-03-25T19:30:11.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.782+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050893177 0 0) 0x7f9c7c1d7880 con 0x7f9c7c0a98f0 2024-03-25T19:30:11.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.782+0000 7f9c72ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.782+0000 7f9c72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.782+0000 7f9c72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.782+0000 7f9c72ffd640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:30:11.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.782+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c7c0abf10 con 0x7f9c7c0a98f0 2024-03-25T19:30:11.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.783+0000 7f9c72ffd640 10 client.?.objecter ms_handle_connect 0x7f9c7c13f710 2024-03-25T19:30:11.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.783+0000 7f9c72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:11.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.792+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478907647 0 0) 0x7f9c7c137c30 con 0x7f9c7c13f710 2024-03-25T19:30:11.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.792+0000 7f9c72ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.792+0000 7f9c72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.792+0000 7f9c72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.792+0000 7f9c72ffd640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:30:11.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.792+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c7c1d7880 con 0x7f9c7c13f710 2024-03-25T19:30:11.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.802+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 128168030 0 0) 0x7f9c7c0abf10 con 0x7f9c7c0a98f0 2024-03-25T19:30:11.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.802+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c7c137c30 con 0x7f9c7c0a98f0 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.802+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9c78001990 con 0x7f9c7c0a98f0 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.802+0000 7f9c72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.802+0000 7f9c72ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.802+0000 7f9c72ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:11.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.802+0000 7f9c72ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:11.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.805+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3403289098 0 0) 0x7f9c7c1d7880 con 0x7f9c7c13f710 2024-03-25T19:30:11.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.805+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c7c0abf10 con 0x7f9c7c13f710 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.805+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9c6c001ae0 con 0x7f9c7c13f710 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.805+0000 7f9c72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.805+0000 7f9c72ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.805+0000 7f9c72ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:11.805 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:11.806 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.809+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3073682778 0 0) 0x7f9c60004eb0 con 0x7f9c7c1430c0 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.809+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c7c1d7880 con 0x7f9c7c1430c0 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.809+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9c74001bd0 con 0x7f9c7c1430c0 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.809+0000 7f9c72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.809+0000 7f9c72ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.809+0000 7f9c72ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:11.809 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:11.810 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:11.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3745006553 0 0) 0x7f9c7c0abf10 con 0x7f9c7c13f710 2024-03-25T19:30:11.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 >> v1:172.21.15.52:6790/0 conn(0x7f9c7c0a98f0 legacy=0x7f9c7c1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 >> v1:172.21.15.52:6789/0 conn(0x7f9c7c1430c0 legacy=0x7f9c7c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c72ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:11.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c72ffd640 1 monclient: found mon.b 2024-03-25T19:30:11.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:11.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c72ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:11.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c1d84b0 con 0x7f9c7c13f710 2024-03-25T19:30:11.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c72ffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:11.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c72ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:11.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:11.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c846e4640 5 monclient: authenticate success, global_id 4351 2024-03-25T19:30:11.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c846e4640 10 monclient: _renew_subs 2024-03-25T19:30:11.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c846e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:11.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c846e4640 1 -- 172.21.15.52:0/2667777983 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7c138d90 con 0x7f9c7c13f710 2024-03-25T19:30:11.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c846e4640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:11.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c846e4640 10 monclient: _renew_subs 2024-03-25T19:30:11.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c846e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:11.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c846e4640 1 -- 172.21.15.52:0/2667777983 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7c1d51e0 con 0x7f9c7c13f710 2024-03-25T19:30:11.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.823+0000 7f9c846e4640 1 librados: init done 2024-03-25T19:30:11.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.824+0000 7f9c846e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:11.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.824+0000 7f9c846e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:11.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.824+0000 7f9c846e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:11.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:11.824+0000 7f9c846e4640 1 -- 172.21.15.52:0/2667777983 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7c1d76e0 con 0x7f9c7c13f710 2024-03-25T19:30:12.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.105+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c6c001210 con 0x7f9c7c13f710 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.105+0000 7f9c72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.105+0000 7f9c80c7d640 4 set_mon_vals no callback set 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.105+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9c6c004a90 con 0x7f9c7c13f710 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.105+0000 7f9c72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.105+0000 7f9c72ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.105+0000 7f9c72ffd640 10 monclient: dump: 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:12.106 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:12.107 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:12.107 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:12.107 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:12.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.106+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f9c6c01aa60 con 0x7f9c7c13f710 2024-03-25T19:30:12.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.107+0000 7f9c82480640 10 monclient: get_auth_request con 0x7f9c6006e070 auth_method 0 2024-03-25T19:30:12.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.107+0000 7f9c72ffd640 10 client.4351.objecter ms_handle_connect 0x7f9c6006e070 2024-03-25T19:30:12.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.113+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f9c6c054680 con 0x7f9c7c13f710 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.113+0000 7f9c72ffd640 10 client.4351.objecter ms_dispatch 0x7f9c7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.113+0000 7f9c72ffd640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.113+0000 7f9c72ffd640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.113+0000 7f9c72ffd640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.114+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c6c04ffc0 con 0x7f9c7c13f710 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.114+0000 7f9c72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:12.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.114+0000 7f9c72ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:12.340 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.340+0000 7f9c846e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:12.340 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.340+0000 7f9c846e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:12.340 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.341+0000 7f9c846e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:12.341 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.341+0000 7f9c846e4640 1 -- 172.21.15.52:0/2667777983 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c7c1d7a10 con 0x7f9c7c13f710 2024-03-25T19:30:12.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.372+0000 7f9c72ffd640 1 -- 172.21.15.52:0/2667777983 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f9c6c0520c0 con 0x7f9c7c13f710 2024-03-25T19:30:12.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.372+0000 7f9c72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:12.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.372+0000 7f9c72ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:12.372 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:30:12.372 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:12.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 10 librados: watch_flush enter 2024-03-25T19:30:12.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 10 librados: watch_flush exit 2024-03-25T19:30:12.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-25T19:30:12.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 10 client.4351.objecter successfully canceled tick 2024-03-25T19:30:12.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 1 -- 172.21.15.52:0/2667777983 >> v1:172.21.15.52:6800/3967427147 conn(0x7f9c6006e070 legacy=0x7f9c60070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:12.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 10 monclient: shutdown 2024-03-25T19:30:12.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:12.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 1 -- 172.21.15.52:0/2667777983 >> v1:172.21.15.179:6789/0 conn(0x7f9c7c13f710 legacy=0x7f9c7c138680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:12.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 1 -- 172.21.15.52:0/2667777983 shutdown_connections 2024-03-25T19:30:12.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 1 -- 172.21.15.52:0/2667777983 >> 172.21.15.52:0/2667777983 conn(0x7f9c7c0b7ea0 msgr2=0x7f9c7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:12.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 1 -- 172.21.15.52:0/2667777983 shutdown_connections 2024-03-25T19:30:12.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 1 -- 172.21.15.52:0/2667777983 wait complete. 2024-03-25T19:30:12.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:12.375+0000 7f9c846e4640 1 librados: shutdown 2024-03-25T19:30:12.386 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:19.193 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:19.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.308+0000 7f5e1a8f1640 10 monclient: get_monmap_and_config 2024-03-25T19:30:19.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.308+0000 7f5e1a8f1640 10 monclient: build_initial_monmap 2024-03-25T19:30:19.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.308+0000 7f5e1a8f1640 10 monclient: monmap: 2024-03-25T19:30:19.308 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:19.308 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:19.308 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:30:19.309070+0000 2024-03-25T19:30:19.308 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:30:19.309070+0000 2024-03-25T19:30:19.308 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:19.308 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:19.308 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:19.309 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:30:19.309 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:30:19.309 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:19.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.309+0000 7f5e1a8f1640 1 Processor -- start 2024-03-25T19:30:19.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.309+0000 7f5e1a8f1640 1 -- start start 2024-03-25T19:30:19.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.309+0000 7f5e1a8f1640 10 monclient: init 2024-03-25T19:30:19.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.310+0000 7f5e1a8f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:19.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.310+0000 7f5e1a8f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:19.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.310+0000 7f5e1a8f1640 10 monclient(hunting): picked mon.noname-a con 0x7f5e140a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:19.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.310+0000 7f5e1a8f1640 10 monclient(hunting): picked mon.noname-b con 0x7f5e1413f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:19.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.310+0000 7f5e1a8f1640 10 monclient(hunting): picked mon.noname-c con 0x7f5e141430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:19.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.310+0000 7f5e1a8f1640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e14142520 con 0x7f5e140a98f0 2024-03-25T19:30:19.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.310+0000 7f5e1a8f1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e140b28a0 con 0x7f5e141430c0 2024-03-25T19:30:19.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.310+0000 7f5e1a8f1640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e140b2a30 con 0x7f5e1413f710 2024-03-25T19:30:19.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.310+0000 7f5e1a8f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:19.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.310+0000 7f5e1a8f1640 10 monclient(hunting): authenticate will time out at 6067.490420s 2024-03-25T19:30:19.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.326+0000 7f5e13fff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f5e140a98f0 0x7f5e140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:45988/0 (socket says 172.21.15.52:45988) 2024-03-25T19:30:19.325 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.326+0000 7f5e13fff640 1 -- 172.21.15.52:0/2454291909 learned_addr learned my addr 172.21.15.52:0/2454291909 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:19.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.340+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852732496 0 0) 0x7f5e140b2a30 con 0x7f5e1413f710 2024-03-25T19:30:19.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.340+0000 7f5e12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.340+0000 7f5e12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.340+0000 7f5e12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.340+0000 7f5e12ffd640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:30:19.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.340+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dfc003520 con 0x7f5e1413f710 2024-03-25T19:30:19.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.343+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206286787 0 0) 0x7f5e140b28a0 con 0x7f5e141430c0 2024-03-25T19:30:19.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.343+0000 7f5e12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.343+0000 7f5e12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.343+0000 7f5e12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.343+0000 7f5e12ffd640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:30:19.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.343+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e140b2a30 con 0x7f5e141430c0 2024-03-25T19:30:19.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.348+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684082624 0 0) 0x7f5e14142520 con 0x7f5e140a98f0 2024-03-25T19:30:19.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.348+0000 7f5e12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.348+0000 7f5e12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.348+0000 7f5e12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.348+0000 7f5e12ffd640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:30:19.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.348+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e140b28a0 con 0x7f5e140a98f0 2024-03-25T19:30:19.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1449408246 0 0) 0x7f5dfc003520 con 0x7f5e1413f710 2024-03-25T19:30:19.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 >> v1:172.21.15.179:6789/0 conn(0x7f5e141430c0 legacy=0x7f5e1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 >> v1:172.21.15.52:6789/0 conn(0x7f5e140a98f0 legacy=0x7f5e140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:19.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 1 monclient: found mon.noname-b 2024-03-25T19:30:19.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:19.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e140b44c0 con 0x7f5e1413f710 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f5e08001690 con 0x7f5e1413f710 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e12ffd640 10 monclient: dump: 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.363 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:19.364 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:19.364 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:19.364 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:19.364 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:19.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e1a8f1640 5 monclient: authenticate success, global_id 4343 2024-03-25T19:30:19.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.362+0000 7f5e1a8f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.375+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e080017f0 con 0x7f5e1413f710 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.375+0000 7f5e12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.375+0000 7f5e12ffd640 1 -- 172.21.15.52:0/2454291909 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f5e08001c00 con 0x7f5e1413f710 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.375+0000 7f5e12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.375+0000 7f5e12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.375+0000 7f5e12ffd640 10 monclient: dump: 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.375 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:19.376 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:19.376 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:19.376 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:19.376 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:19.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.375+0000 7f5e1a8f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:19.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.375+0000 7f5e1a8f1640 4 set_mon_vals no callback set 2024-03-25T19:30:19.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.376+0000 7f5e1a8f1640 10 monclient: shutdown 2024-03-25T19:30:19.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.376+0000 7f5e1a8f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:19.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.376+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/2454291909 >> v1:172.21.15.52:6790/0 conn(0x7f5e1413f710 legacy=0x7f5e141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.376+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/2454291909 shutdown_connections 2024-03-25T19:30:19.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.376+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/2454291909 >> 172.21.15.52:0/2454291909 conn(0x7f5e140b7ea0 msgr2=0x7f5e140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:19.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.376+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/2454291909 shutdown_connections 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.376+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/2454291909 wait complete. 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.377+0000 7f5e1a8f1640 10 monclient: build_initial_monmap 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.377+0000 7f5e1a8f1640 10 monclient: monmap: 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:19.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.378+0000 7f5e1a8f1640 1 librados: starting msgr at 2024-03-25T19:30:19.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.378+0000 7f5e1a8f1640 1 librados: starting objecter 2024-03-25T19:30:19.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.378+0000 7f5e1a8f1640 1 Processor -- start 2024-03-25T19:30:19.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.378+0000 7f5e1a8f1640 1 -- start start 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.378+0000 7f5e1a8f1640 1 librados: setting wanted keys 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.378+0000 7f5e1a8f1640 1 librados: calling monclient init 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.378+0000 7f5e1a8f1640 10 monclient: init 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.379+0000 7f5e1a8f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.379+0000 7f5e1a8f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.379+0000 7f5e1a8f1640 10 monclient(hunting): picked mon.noname-b con 0x7f5e140a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.379+0000 7f5e1a8f1640 10 monclient(hunting): picked mon.noname-a con 0x7f5e1413f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.379+0000 7f5e1a8f1640 10 monclient(hunting): picked mon.noname-c con 0x7f5e141430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.379+0000 7f5e1a8f1640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e14139070 con 0x7f5e1413f710 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.379+0000 7f5e1a8f1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e140ab9c0 con 0x7f5e140a98f0 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.379+0000 7f5e1a8f1640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e140aca40 con 0x7f5e141430c0 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.379+0000 7f5e1a8f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:19.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.379+0000 7f5e1a8f1640 10 monclient(hunting): authenticate will time out at 6067.559569s 2024-03-25T19:30:19.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.389+0000 7f5e18e8e640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f5e141430c0 0x7f5e141e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:41962/0 (socket says 172.21.15.52:41962) 2024-03-25T19:30:19.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.389+0000 7f5e18e8e640 1 -- 172.21.15.52:0/926850413 learned_addr learned my addr 172.21.15.52:0/926850413 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:19.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.390+0000 7f5e10ff9640 10 client.?.objecter ms_handle_connect 0x7f5e141430c0 2024-03-25T19:30:19.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.390+0000 7f5e10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:19.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.392+0000 7f5e10ff9640 10 client.?.objecter ms_handle_connect 0x7f5e1413f710 2024-03-25T19:30:19.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.392+0000 7f5e10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:19.392 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.393+0000 7f5e10ff9640 10 client.?.objecter ms_handle_connect 0x7f5e140a98f0 2024-03-25T19:30:19.392 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.393+0000 7f5e10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:19.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.399+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911515280 0 0) 0x7f5e140ab9c0 con 0x7f5e140a98f0 2024-03-25T19:30:19.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.399+0000 7f5e10ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.399+0000 7f5e10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.399+0000 7f5e10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.399+0000 7f5e10ff9640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:30:19.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.399+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e00004eb0 con 0x7f5e140a98f0 2024-03-25T19:30:19.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842467707 0 0) 0x7f5e140aca40 con 0x7f5e141430c0 2024-03-25T19:30:19.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:30:19.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e140ab9c0 con 0x7f5e141430c0 2024-03-25T19:30:19.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870434758 0 0) 0x7f5e14139070 con 0x7f5e1413f710 2024-03-25T19:30:19.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:30:19.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:19.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:19.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:30:19.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.404+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e140aca40 con 0x7f5e1413f710 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.424+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3871474362 0 0) 0x7f5e140ab9c0 con 0x7f5e141430c0 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.424+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e14139070 con 0x7f5e141430c0 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.424+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f5e0c001c10 con 0x7f5e141430c0 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.424+0000 7f5e10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.424+0000 7f5e10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.424+0000 7f5e10ff9640 10 monclient(hunting): dump: 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:19.424 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:19.425 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:19.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.424+0000 7f5e10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:19.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.426+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3038960362 0 0) 0x7f5e00004eb0 con 0x7f5e140a98f0 2024-03-25T19:30:19.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.426+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e140ab9c0 con 0x7f5e140a98f0 2024-03-25T19:30:19.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.426+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f5e040018f0 con 0x7f5e140a98f0 2024-03-25T19:30:19.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.426+0000 7f5e10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.426+0000 7f5e10ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:19.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.426+0000 7f5e10ff9640 10 monclient(hunting): dump: 2024-03-25T19:30:19.426 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:19.426 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:19.427 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.427 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.427 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.427 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:19.427 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:19.427 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:19.427 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:19.427 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:19.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1227406987 0 0) 0x7f5e140aca40 con 0x7f5e1413f710 2024-03-25T19:30:19.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e00004eb0 con 0x7f5e1413f710 2024-03-25T19:30:19.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f5e08001b00 con 0x7f5e1413f710 2024-03-25T19:30:19.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 10 monclient(hunting): dump: 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3513735295 0 0) 0x7f5e14139070 con 0x7f5e141430c0 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 >> v1:172.21.15.179:6789/0 conn(0x7f5e140a98f0 legacy=0x7f5e14138830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 >> v1:172.21.15.52:6789/0 conn(0x7f5e1413f710 legacy=0x7f5e140ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 1 monclient: found mon.c 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e140abc30 con 0x7f5e141430c0 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 10 monclient: _finish_auth 0 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e1a8f1640 5 monclient: authenticate success, global_id 4346 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e1a8f1640 10 monclient: _renew_subs 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e1a8f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/926850413 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e140abf10 con 0x7f5e141430c0 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e1a8f1640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e1a8f1640 10 monclient: _renew_subs 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e1a8f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:30:19.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/926850413 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e14137e60 con 0x7f5e141430c0 2024-03-25T19:30:19.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.439+0000 7f5e1a8f1640 1 librados: init done 2024-03-25T19:30:19.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.440+0000 7f5e1a8f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:19.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.440+0000 7f5e1a8f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:19.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.440+0000 7f5e1a8f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:30:19.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.440+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/926850413 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e140627b0 con 0x7f5e141430c0 2024-03-25T19:30:19.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.756+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e0c003990 con 0x7f5e141430c0 2024-03-25T19:30:19.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.756+0000 7f5e10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:19.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.756+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f5e0c002100 con 0x7f5e141430c0 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.756+0000 7f5e10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.756+0000 7f5e10ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.756+0000 7f5e10ff9640 10 monclient: dump: 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:19.756 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:19.757 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:19.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.756+0000 7f5e127fc640 4 set_mon_vals no callback set 2024-03-25T19:30:19.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.756+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f5e0c003e60 con 0x7f5e141430c0 2024-03-25T19:30:19.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.757+0000 7f5e13fff640 10 monclient: get_auth_request con 0x7f5e0006e0f0 auth_method 0 2024-03-25T19:30:19.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.757+0000 7f5e10ff9640 10 client.4346.objecter ms_handle_connect 0x7f5e0006e0f0 2024-03-25T19:30:19.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.763+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f5e0c0549e0 con 0x7f5e141430c0 2024-03-25T19:30:19.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.763+0000 7f5e10ff9640 10 client.4346.objecter ms_dispatch 0x7f5e14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:19.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.763+0000 7f5e10ff9640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:19.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.763+0000 7f5e10ff9640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:19.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.764+0000 7f5e10ff9640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-25T19:30:19.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.764+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e0c0504f0 con 0x7f5e141430c0 2024-03-25T19:30:19.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.764+0000 7f5e10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:19.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.764+0000 7f5e10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:19.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.985+0000 7f5e1a8f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:19.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.985+0000 7f5e1a8f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:19.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.985+0000 7f5e1a8f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:30:19.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:19.985+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/926850413 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e1413e250 con 0x7f5e141430c0 2024-03-25T19:30:20.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.031+0000 7f5e10ff9640 1 -- 172.21.15.52:0/926850413 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f5e0c052420 con 0x7f5e141430c0 2024-03-25T19:30:20.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.031+0000 7f5e10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:20.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.031+0000 7f5e10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:20.032 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:30:20.032 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:20.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 10 librados: watch_flush enter 2024-03-25T19:30:20.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 10 librados: watch_flush exit 2024-03-25T19:30:20.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-25T19:30:20.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 10 client.4346.objecter successfully canceled tick 2024-03-25T19:30:20.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/926850413 >> v1:172.21.15.52:6800/3967427147 conn(0x7f5e0006e0f0 legacy=0x7f5e000705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:20.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 10 monclient: shutdown 2024-03-25T19:30:20.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:20.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/926850413 >> v1:172.21.15.52:6790/0 conn(0x7f5e141430c0 legacy=0x7f5e141e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:20.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/926850413 shutdown_connections 2024-03-25T19:30:20.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/926850413 >> 172.21.15.52:0/926850413 conn(0x7f5e140b7ea0 msgr2=0x7f5e1413c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:20.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/926850413 shutdown_connections 2024-03-25T19:30:20.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 1 -- 172.21.15.52:0/926850413 wait complete. 2024-03-25T19:30:20.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:20.034+0000 7f5e1a8f1640 1 librados: shutdown 2024-03-25T19:30:20.046 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:26.263 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:03:19.861 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:26.279 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:03:19.878 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:26.311 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:03:19.910 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:26.333 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:03:19.875 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:26.342 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:03:19.884 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:26.345 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:03:19.944 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:26.352 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:03:19.951 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:26.363 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:03:19.905 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:26.372 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:03:19.971 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:26.852 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:26.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.969+0000 7f9d6710e640 10 monclient: get_monmap_and_config 2024-03-25T19:30:26.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.969+0000 7f9d6710e640 10 monclient: build_initial_monmap 2024-03-25T19:30:26.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.969+0000 7f9d6710e640 10 monclient: monmap: 2024-03-25T19:30:26.969 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:26.969 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:26.969 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:30:26.970451+0000 2024-03-25T19:30:26.969 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:30:26.970451+0000 2024-03-25T19:30:26.970 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:26.970 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:26.970 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:26.970 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:30:26.970 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:30:26.970 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:26.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.970+0000 7f9d6710e640 1 Processor -- start 2024-03-25T19:30:26.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.970+0000 7f9d6710e640 1 -- start start 2024-03-25T19:30:26.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.970+0000 7f9d6710e640 10 monclient: init 2024-03-25T19:30:26.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.971+0000 7f9d6710e640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:26.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.971+0000 7f9d6710e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:26.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.971+0000 7f9d6710e640 10 monclient(hunting): picked mon.noname-a con 0x7f9d600a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:26.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.971+0000 7f9d6710e640 10 monclient(hunting): picked mon.noname-b con 0x7f9d6013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:26.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.971+0000 7f9d6710e640 10 monclient(hunting): picked mon.noname-c con 0x7f9d601430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:26.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.971+0000 7f9d6710e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d60142520 con 0x7f9d600a98f0 2024-03-25T19:30:26.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.971+0000 7f9d6710e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d600b28a0 con 0x7f9d601430c0 2024-03-25T19:30:26.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.971+0000 7f9d6710e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d600b2a30 con 0x7f9d6013f710 2024-03-25T19:30:26.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.971+0000 7f9d6710e640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:26.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.971+0000 7f9d6710e640 10 monclient(hunting): authenticate will time out at 6075.151471s 2024-03-25T19:30:26.979 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.980+0000 7f9d5ffff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f9d6013f710 0x7f9d601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:39950/0 (socket says 172.21.15.52:39950) 2024-03-25T19:30:27.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.980+0000 7f9d5ffff640 1 -- 172.21.15.52:0/2768628823 learned_addr learned my addr 172.21.15.52:0/2768628823 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:27.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.991+0000 7f9d5f7fe640 1 -- 172.21.15.52:0/2768628823 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001842057 0 0) 0x7f9d600b2a30 con 0x7f9d6013f710 2024-03-25T19:30:27.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.992+0000 7f9d5f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.992+0000 7f9d5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.992+0000 7f9d5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.992+0000 7f9d5f7fe640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.992+0000 7f9d5f7fe640 1 -- 172.21.15.52:0/2768628823 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d50003520 con 0x7f9d6013f710 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.999+0000 7f9d5f7fe640 1 -- 172.21.15.52:0/2768628823 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384743093 0 0) 0x7f9d600b28a0 con 0x7f9d601430c0 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.999+0000 7f9d5f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.999+0000 7f9d5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.999+0000 7f9d5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.999+0000 7f9d5f7fe640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:26.999+0000 7f9d5f7fe640 1 -- 172.21.15.52:0/2768628823 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d600b2a30 con 0x7f9d601430c0 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d5f7fe640 1 -- 172.21.15.52:0/2768628823 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2809246523 0 0) 0x7f9d50003520 con 0x7f9d6013f710 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d5f7fe640 1 -- 172.21.15.52:0/2768628823 >> v1:172.21.15.179:6789/0 conn(0x7f9d601430c0 legacy=0x7f9d6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d5f7fe640 1 -- 172.21.15.52:0/2768628823 >> v1:172.21.15.52:6789/0 conn(0x7f9d600a98f0 legacy=0x7f9d600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d5f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d5f7fe640 1 monclient: found mon.noname-b 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d5f7fe640 1 -- 172.21.15.52:0/2768628823 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d600b44c0 con 0x7f9d6013f710 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d5f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d5f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d6710e640 5 monclient: authenticate success, global_id 4349 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.002+0000 7f9d6710e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.003+0000 7f9d5f7fe640 1 -- 172.21.15.52:0/2768628823 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9d4c001650 con 0x7f9d6013f710 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.003+0000 7f9d5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:27.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.003+0000 7f9d5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.003+0000 7f9d5f7fe640 10 monclient: dump: 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.003+0000 7f9d6710e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.028+0000 7f9d5f7fe640 1 -- 172.21.15.52:0/2768628823 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d4c0018a0 con 0x7f9d6013f710 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.028+0000 7f9d5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:27.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.028+0000 7f9d5f7fe640 1 -- 172.21.15.52:0/2768628823 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9d4c001c30 con 0x7f9d6013f710 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.028+0000 7f9d5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.028+0000 7f9d5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.028+0000 7f9d5f7fe640 10 monclient: dump: 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.028+0000 7f9d6710e640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:27.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.028+0000 7f9d6710e640 4 set_mon_vals no callback set 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.029+0000 7f9d6710e640 10 monclient: shutdown 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.029+0000 7f9d6710e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.029+0000 7f9d6710e640 1 -- 172.21.15.52:0/2768628823 >> v1:172.21.15.52:6790/0 conn(0x7f9d6013f710 legacy=0x7f9d601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.029+0000 7f9d6710e640 1 -- 172.21.15.52:0/2768628823 shutdown_connections 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.029+0000 7f9d6710e640 1 -- 172.21.15.52:0/2768628823 >> 172.21.15.52:0/2768628823 conn(0x7f9d600b7ea0 msgr2=0x7f9d600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.029+0000 7f9d6710e640 1 -- 172.21.15.52:0/2768628823 shutdown_connections 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.029+0000 7f9d6710e640 1 -- 172.21.15.52:0/2768628823 wait complete. 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.030+0000 7f9d6710e640 10 monclient: build_initial_monmap 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.030+0000 7f9d6710e640 10 monclient: monmap: 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:27.052 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.030+0000 7f9d6710e640 1 librados: starting msgr at 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.030+0000 7f9d6710e640 1 librados: starting objecter 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.030+0000 7f9d6710e640 1 Processor -- start 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.030+0000 7f9d6710e640 1 -- start start 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.030+0000 7f9d6710e640 1 librados: setting wanted keys 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.030+0000 7f9d6710e640 1 librados: calling monclient init 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.030+0000 7f9d6710e640 10 monclient: init 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.031+0000 7f9d6710e640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.031+0000 7f9d6710e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.031+0000 7f9d6710e640 10 monclient(hunting): picked mon.noname-c con 0x7f9d600a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.031+0000 7f9d6710e640 10 monclient(hunting): picked mon.noname-a con 0x7f9d6013f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:27.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.031+0000 7f9d6710e640 10 monclient(hunting): picked mon.noname-b con 0x7f9d601430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.031+0000 7f9d6710e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d60138db0 con 0x7f9d6013f710 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.031+0000 7f9d6710e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d60137340 con 0x7f9d601430c0 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.031+0000 7f9d6710e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d601369c0 con 0x7f9d600a98f0 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.031+0000 7f9d6710e640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.031+0000 7f9d6710e640 10 monclient(hunting): authenticate will time out at 6075.211788s 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.038+0000 7f9d5ffff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f9d6013f710 0x7f9d60137860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:33282/0 (socket says 172.21.15.52:33282) 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.038+0000 7f9d5ffff640 1 -- 172.21.15.52:0/1185697377 learned_addr learned my addr 172.21.15.52:0/1185697377 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.039+0000 7f9d5d7fa640 10 client.?.objecter ms_handle_connect 0x7f9d6013f710 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.039+0000 7f9d5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.047+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28355595 0 0) 0x7f9d60138db0 con 0x7f9d6013f710 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.047+0000 7f9d5d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.047+0000 7f9d5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.047+0000 7f9d5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:27.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.047+0000 7f9d5d7fa640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:30:27.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.047+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d44004eb0 con 0x7f9d6013f710 2024-03-25T19:30:27.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.052+0000 7f9d5d7fa640 10 client.?.objecter ms_handle_connect 0x7f9d601430c0 2024-03-25T19:30:27.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.052+0000 7f9d5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:27.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.058+0000 7f9d5d7fa640 10 client.?.objecter ms_handle_connect 0x7f9d600a98f0 2024-03-25T19:30:27.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.058+0000 7f9d5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:27.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2247903791 0 0) 0x7f9d44004eb0 con 0x7f9d6013f710 2024-03-25T19:30:27.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d60138db0 con 0x7f9d6013f710 2024-03-25T19:30:27.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9d4c001bb0 con 0x7f9d6013f710 2024-03-25T19:30:27.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 10 monclient(hunting): dump: 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088746728 0 0) 0x7f9d60137340 con 0x7f9d601430c0 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:27.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:27.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:27.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:30:27.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.064+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d44004eb0 con 0x7f9d601430c0 2024-03-25T19:30:27.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.074+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 656013932 0 0) 0x7f9d44004eb0 con 0x7f9d601430c0 2024-03-25T19:30:27.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.074+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d60137340 con 0x7f9d601430c0 2024-03-25T19:30:27.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.074+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9d58001b50 con 0x7f9d601430c0 2024-03-25T19:30:27.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.074+0000 7f9d5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:27.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.074+0000 7f9d5d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:27.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.074+0000 7f9d5d7fa640 10 monclient(hunting): dump: 2024-03-25T19:30:27.074 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:27.074 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:27.074 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:27.074 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:27.074 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:27.074 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:27.074 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:27.074 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:27.074 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:27.074 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:27.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2862075043 0 0) 0x7f9d60138db0 con 0x7f9d6013f710 2024-03-25T19:30:27.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 >> v1:172.21.15.52:6790/0 conn(0x7f9d600a98f0 legacy=0x7f9d601dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 >> v1:172.21.15.179:6789/0 conn(0x7f9d601430c0 legacy=0x7f9d60139440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d5d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:30:27.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d5d7fa640 1 monclient: found mon.a 2024-03-25T19:30:27.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d5d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:27.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d5d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:30:27.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d600b7190 con 0x7f9d6013f710 2024-03-25T19:30:27.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d5d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:30:27.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d5d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:30:27.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:27.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d6710e640 5 monclient: authenticate success, global_id 4353 2024-03-25T19:30:27.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d6710e640 10 monclient: _renew_subs 2024-03-25T19:30:27.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d6710e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:30:27.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d6710e640 1 -- 172.21.15.52:0/1185697377 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d600b7440 con 0x7f9d6013f710 2024-03-25T19:30:27.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d6710e640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:27.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d6710e640 10 monclient: _renew_subs 2024-03-25T19:30:27.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d6710e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:30:27.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d6710e640 1 -- 172.21.15.52:0/1185697377 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d601331e0 con 0x7f9d6013f710 2024-03-25T19:30:27.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.078+0000 7f9d6710e640 1 librados: init done 2024-03-25T19:30:27.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.079+0000 7f9d6710e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:27.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.079+0000 7f9d6710e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:27.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.079+0000 7f9d6710e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:30:27.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.079+0000 7f9d6710e640 1 -- 172.21.15.52:0/1185697377 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d600627b0 con 0x7f9d6013f710 2024-03-25T19:30:27.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.397+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d4c001d10 con 0x7f9d6013f710 2024-03-25T19:30:27.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.397+0000 7f9d5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:27.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.397+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9d4c004d90 con 0x7f9d6013f710 2024-03-25T19:30:27.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.397+0000 7f9d5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:27.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.397+0000 7f9d5d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:27.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.397+0000 7f9d5d7fa640 10 monclient: dump: 2024-03-25T19:30:27.397 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:27.397 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:27.397 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:27.397 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:27.398 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:27.398 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:27.398 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:27.398 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:27.398 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:27.398 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:27.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.397+0000 7f9d5effd640 4 set_mon_vals no callback set 2024-03-25T19:30:27.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.398+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f9d4c01ae20 con 0x7f9d6013f710 2024-03-25T19:30:27.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.398+0000 7f9d64eaa640 10 monclient: get_auth_request con 0x7f9d4406de00 auth_method 0 2024-03-25T19:30:27.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.399+0000 7f9d5d7fa640 10 client.4353.objecter ms_handle_connect 0x7f9d4406de00 2024-03-25T19:30:27.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.405+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f9d4c054a00 con 0x7f9d6013f710 2024-03-25T19:30:27.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.405+0000 7f9d5d7fa640 10 client.4353.objecter ms_dispatch 0x7f9d60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:27.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.405+0000 7f9d5d7fa640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:27.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.405+0000 7f9d5d7fa640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:27.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.405+0000 7f9d5d7fa640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-25T19:30:27.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.405+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d4c050340 con 0x7f9d6013f710 2024-03-25T19:30:27.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.405+0000 7f9d5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:27.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.405+0000 7f9d5d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:27.680 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.681+0000 7f9d6710e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:27.680 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.681+0000 7f9d6710e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:27.680 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.681+0000 7f9d6710e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:30:27.680 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.681+0000 7f9d6710e640 1 -- 172.21.15.52:0/1185697377 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d60142ad0 con 0x7f9d6013f710 2024-03-25T19:30:27.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.713+0000 7f9d5d7fa640 1 -- 172.21.15.52:0/1185697377 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f9d4c052440 con 0x7f9d6013f710 2024-03-25T19:30:27.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.714+0000 7f9d5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:27.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.714+0000 7f9d5d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:27.713 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:30:27.713 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:27.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 10 librados: watch_flush enter 2024-03-25T19:30:27.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 10 librados: watch_flush exit 2024-03-25T19:30:27.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-25T19:30:27.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 10 client.4353.objecter successfully canceled tick 2024-03-25T19:30:27.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 1 -- 172.21.15.52:0/1185697377 >> v1:172.21.15.52:6800/3967427147 conn(0x7f9d4406de00 legacy=0x7f9d440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 10 monclient: shutdown 2024-03-25T19:30:27.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:27.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 1 -- 172.21.15.52:0/1185697377 >> v1:172.21.15.52:6789/0 conn(0x7f9d6013f710 legacy=0x7f9d60137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:27.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 1 -- 172.21.15.52:0/1185697377 shutdown_connections 2024-03-25T19:30:27.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 1 -- 172.21.15.52:0/1185697377 >> 172.21.15.52:0/1185697377 conn(0x7f9d600b7ea0 msgr2=0x7f9d6013c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:27.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 1 -- 172.21.15.52:0/1185697377 shutdown_connections 2024-03-25T19:30:27.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 1 -- 172.21.15.52:0/1185697377 wait complete. 2024-03-25T19:30:27.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:27.717+0000 7f9d6710e640 1 librados: shutdown 2024-03-25T19:30:27.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:34.535 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:34.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.655+0000 7f771f563640 10 monclient: get_monmap_and_config 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.655+0000 7f771f563640 10 monclient: build_initial_monmap 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.655+0000 7f771f563640 10 monclient: monmap: 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:30:34.655696+0000 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:30:34.655696+0000 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:30:34.655 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.655+0000 7f771f563640 1 Processor -- start 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 1 -- start start 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 10 monclient: init 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 10 monclient(hunting): picked mon.noname-c con 0x7f771806ea40 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 10 monclient(hunting): picked mon.noname-a con 0x7f7718071800 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 10 monclient(hunting): picked mon.noname-b con 0x7f7718071c40 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771806f1f0 con 0x7f7718071800 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7718070d00 con 0x7f771806ea40 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77180720b0 con 0x7f7718071c40 2024-03-25T19:30:34.657 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:34.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.656+0000 7f771f563640 10 monclient(hunting): authenticate will time out at 6082.836897s 2024-03-25T19:30:34.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.672+0000 7f771d2ff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f771806ea40 0x7f771813ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:37618/0 (socket says 172.21.15.52:37618) 2024-03-25T19:30:34.672 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.672+0000 7f771d2ff640 1 -- 172.21.15.52:0/1973001349 learned_addr learned my addr 172.21.15.52:0/1973001349 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:34.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.683+0000 7f770ffff640 1 -- 172.21.15.52:0/1973001349 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569559064 0 0) 0x7f7718070d00 con 0x7f771806ea40 2024-03-25T19:30:34.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.683+0000 7f770ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.683+0000 7f770ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.683+0000 7f770ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.683+0000 7f770ffff640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:30:34.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.683+0000 7f770ffff640 1 -- 172.21.15.52:0/1973001349 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7704003520 con 0x7f771806ea40 2024-03-25T19:30:34.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.686+0000 7f770ffff640 1 -- 172.21.15.52:0/1973001349 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187654411 0 0) 0x7f77180720b0 con 0x7f7718071c40 2024-03-25T19:30:34.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.686+0000 7f770ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.686+0000 7f770ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.686+0000 7f770ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.686+0000 7f770ffff640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:30:34.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.686+0000 7f770ffff640 1 -- 172.21.15.52:0/1973001349 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7718070d00 con 0x7f7718071c40 2024-03-25T19:30:34.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 1 -- 172.21.15.52:0/1973001349 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 884436166 0 0) 0x7f7704003520 con 0x7f771806ea40 2024-03-25T19:30:34.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 1 -- 172.21.15.52:0/1973001349 >> v1:172.21.15.52:6790/0 conn(0x7f7718071c40 legacy=0x7f771814a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 1 -- 172.21.15.52:0/1973001349 >> v1:172.21.15.52:6789/0 conn(0x7f7718071800 legacy=0x7f7718147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:34.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 1 monclient: found mon.noname-c 2024-03-25T19:30:34.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:34.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:30:34.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 1 -- 172.21.15.52:0/1973001349 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7718072780 con 0x7f771806ea40 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 10 monclient: _finish_auth 0 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 1 -- 172.21.15.52:0/1973001349 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7714001690 con 0x7f771806ea40 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.687+0000 7f770ffff640 10 monclient: dump: 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:34.688 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:34.689 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:34.689 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:34.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.688+0000 7f771f563640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:30:34.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.688+0000 7f771f563640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:34.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.710+0000 7f770ffff640 1 -- 172.21.15.52:0/1973001349 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7714001870 con 0x7f771806ea40 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.710+0000 7f770ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.710+0000 7f770ffff640 1 -- 172.21.15.52:0/1973001349 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7714001c80 con 0x7f771806ea40 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.710+0000 7f770ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.710+0000 7f770ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.710+0000 7f770ffff640 10 monclient: dump: 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:34.710 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:34.711 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:34.711 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:34.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.710+0000 7f771f563640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:34.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.710+0000 7f771f563640 4 set_mon_vals no callback set 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.711+0000 7f771f563640 10 monclient: shutdown 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.711+0000 7f771f563640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.711+0000 7f771f563640 1 -- 172.21.15.52:0/1973001349 >> v1:172.21.15.179:6789/0 conn(0x7f771806ea40 legacy=0x7f771813ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.711+0000 7f771f563640 1 -- 172.21.15.52:0/1973001349 shutdown_connections 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.711+0000 7f771f563640 1 -- 172.21.15.52:0/1973001349 >> 172.21.15.52:0/1973001349 conn(0x7f771806b630 msgr2=0x7f77180b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.711+0000 7f771f563640 1 -- 172.21.15.52:0/1973001349 shutdown_connections 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.711+0000 7f771f563640 1 -- 172.21.15.52:0/1973001349 wait complete. 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.712+0000 7f771f563640 10 monclient: build_initial_monmap 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.712+0000 7f771f563640 10 monclient: monmap: 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:34.713 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.712+0000 7f771f563640 1 librados: starting msgr at 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.712+0000 7f771f563640 1 librados: starting objecter 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.712+0000 7f771f563640 1 Processor -- start 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.712+0000 7f771f563640 1 -- start start 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.712+0000 7f771f563640 1 librados: setting wanted keys 2024-03-25T19:30:34.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.712+0000 7f771f563640 1 librados: calling monclient init 2024-03-25T19:30:34.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.712+0000 7f771f563640 10 monclient: init 2024-03-25T19:30:34.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.713+0000 7f771f563640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:34.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.713+0000 7f771f563640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:34.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.713+0000 7f771f563640 10 monclient(hunting): picked mon.noname-a con 0x7f771806ea40 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:34.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.713+0000 7f771f563640 10 monclient(hunting): picked mon.noname-c con 0x7f7718071800 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:34.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.713+0000 7f771f563640 10 monclient(hunting): picked mon.noname-b con 0x7f7718071c40 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:34.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.713+0000 7f771f563640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77181c9d10 con 0x7f771806ea40 2024-03-25T19:30:34.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.713+0000 7f771f563640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7718138c90 con 0x7f7718071c40 2024-03-25T19:30:34.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.713+0000 7f771f563640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77181cb2c0 con 0x7f7718071800 2024-03-25T19:30:34.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.713+0000 7f771f563640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:34.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.713+0000 7f771f563640 10 monclient(hunting): authenticate will time out at 6082.893483s 2024-03-25T19:30:34.721 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.721+0000 7f771cafe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f7718071800 0x7f7718137e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:45798/0 (socket says 172.21.15.52:45798) 2024-03-25T19:30:34.721 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.722+0000 7f771cafe640 1 -- 172.21.15.52:0/3057814190 learned_addr learned my addr 172.21.15.52:0/3057814190 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:34.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.725+0000 7f770dffb640 10 client.?.objecter ms_handle_connect 0x7f771806ea40 2024-03-25T19:30:34.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.725+0000 7f770dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:34.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.726+0000 7f770dffb640 10 client.?.objecter ms_handle_connect 0x7f7718071800 2024-03-25T19:30:34.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.726+0000 7f770dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:34.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.732+0000 7f770dffb640 10 client.?.objecter ms_handle_connect 0x7f7718071c40 2024-03-25T19:30:34.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.732+0000 7f770dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:34.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.743+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925855775 0 0) 0x7f77181cb2c0 con 0x7f7718071800 2024-03-25T19:30:34.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.744+0000 7f770dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.744+0000 7f770dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.744+0000 7f770dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.744+0000 7f770dffb640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:30:34.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.744+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76fc004eb0 con 0x7f7718071800 2024-03-25T19:30:34.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.744+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770287657 0 0) 0x7f77181c9d10 con 0x7f771806ea40 2024-03-25T19:30:34.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.744+0000 7f770dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.744+0000 7f770dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.744+0000 7f770dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.744+0000 7f770dffb640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:30:34.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.744+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77181cb2c0 con 0x7f771806ea40 2024-03-25T19:30:34.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.749+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785566152 0 0) 0x7f7718138c90 con 0x7f7718071c40 2024-03-25T19:30:34.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.749+0000 7f770dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:34.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.749+0000 7f770dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:34.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.749+0000 7f770dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:34.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.749+0000 7f770dffb640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:30:34.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.749+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77181c9d10 con 0x7f7718071c40 2024-03-25T19:30:34.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.758+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 479579273 0 0) 0x7f77181c9d10 con 0x7f7718071c40 2024-03-25T19:30:34.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.758+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7718138c90 con 0x7f7718071c40 2024-03-25T19:30:34.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.758+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7710001b90 con 0x7f7718071c40 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.758+0000 7f770dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.758+0000 7f770dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.758+0000 7f770dffb640 10 monclient(hunting): dump: 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:34.758 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:34.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.758+0000 7f770dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:34.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.762+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1568132207 0 0) 0x7f76fc004eb0 con 0x7f7718071800 2024-03-25T19:30:34.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.762+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77181c9d10 con 0x7f7718071800 2024-03-25T19:30:34.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.762+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f77080019d0 con 0x7f7718071800 2024-03-25T19:30:34.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.762+0000 7f770dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.762+0000 7f770dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.762+0000 7f770dffb640 10 monclient(hunting): dump: 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:34.762 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:34.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1658833762 0 0) 0x7f77181c9d10 con 0x7f7718071800 2024-03-25T19:30:34.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 >> v1:172.21.15.179:6789/0 conn(0x7f7718071c40 legacy=0x7f7718138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 >> v1:172.21.15.52:6789/0 conn(0x7f771806ea40 legacy=0x7f77181d7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f770dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:30:34.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f770dffb640 1 monclient: found mon.c 2024-03-25T19:30:34.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f770dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:34.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f770dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:30:34.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77181cb5a0 con 0x7f7718071800 2024-03-25T19:30:34.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f770dffb640 10 monclient: _finish_auth 0 2024-03-25T19:30:34.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f770dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:30:34.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f770dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:34.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f771f563640 5 monclient: authenticate success, global_id 4358 2024-03-25T19:30:34.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f771f563640 10 monclient: _renew_subs 2024-03-25T19:30:34.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f771f563640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:30:34.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f771f563640 1 -- 172.21.15.52:0/3057814190 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77181d7b80 con 0x7f7718071800 2024-03-25T19:30:34.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f771f563640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:34.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f771f563640 10 monclient: _renew_subs 2024-03-25T19:30:34.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f771f563640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:30:34.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f771f563640 1 -- 172.21.15.52:0/3057814190 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77181c98a0 con 0x7f7718071800 2024-03-25T19:30:34.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.768+0000 7f771f563640 1 librados: init done 2024-03-25T19:30:34.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.769+0000 7f771f563640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:34.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.769+0000 7f771f563640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:34.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.769+0000 7f771f563640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:30:34.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:34.769+0000 7f771f563640 1 -- 172.21.15.52:0/3057814190 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77181c9ec0 con 0x7f7718071800 2024-03-25T19:30:35.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.095+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7708003610 con 0x7f7718071800 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.095+0000 7f770dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.095+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7708003880 con 0x7f7718071800 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.095+0000 7f770dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.095+0000 7f770dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.095+0000 7f770dffb640 10 monclient: dump: 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:35.095 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:35.096 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:35.096 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:35.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.095+0000 7f770f7fe640 4 set_mon_vals no callback set 2024-03-25T19:30:35.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.096+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f7708003aa0 con 0x7f7718071800 2024-03-25T19:30:35.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.096+0000 7f771d2ff640 10 monclient: get_auth_request con 0x7f76fc06dfb0 auth_method 0 2024-03-25T19:30:35.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.096+0000 7f770dffb640 10 client.4358.objecter ms_handle_connect 0x7f76fc06dfb0 2024-03-25T19:30:35.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.102+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f7708054660 con 0x7f7718071800 2024-03-25T19:30:35.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.103+0000 7f770dffb640 10 client.4358.objecter ms_dispatch 0x7f7718000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:35.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.103+0000 7f770dffb640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:35.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.103+0000 7f770dffb640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:35.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.103+0000 7f770dffb640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-25T19:30:35.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.103+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7708050170 con 0x7f7718071800 2024-03-25T19:30:35.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.103+0000 7f770dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:35.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.103+0000 7f770dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:35.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.333+0000 7f771f563640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:35.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.333+0000 7f771f563640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:35.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.333+0000 7f771f563640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:30:35.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.333+0000 7f771f563640 1 -- 172.21.15.52:0/3057814190 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77180627b0 con 0x7f7718071800 2024-03-25T19:30:35.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.380+0000 7f770dffb640 1 -- 172.21.15.52:0/3057814190 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f77080520a0 con 0x7f7718071800 2024-03-25T19:30:35.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.380+0000 7f770dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:35.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.380+0000 7f770dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:35.380 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:30:35.380 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:35.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.382+0000 7f771f563640 10 librados: watch_flush enter 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.382+0000 7f771f563640 10 librados: watch_flush exit 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.382+0000 7f771f563640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.382+0000 7f771f563640 10 client.4358.objecter successfully canceled tick 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.383+0000 7f771f563640 1 -- 172.21.15.52:0/3057814190 >> v1:172.21.15.52:6800/3967427147 conn(0x7f76fc06dfb0 legacy=0x7f76fc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.383+0000 7f771f563640 10 monclient: shutdown 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.383+0000 7f771f563640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.383+0000 7f771f563640 1 -- 172.21.15.52:0/3057814190 >> v1:172.21.15.52:6790/0 conn(0x7f7718071800 legacy=0x7f7718137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.383+0000 7f771f563640 1 -- 172.21.15.52:0/3057814190 shutdown_connections 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.383+0000 7f771f563640 1 -- 172.21.15.52:0/3057814190 >> 172.21.15.52:0/3057814190 conn(0x7f771806b630 msgr2=0x7f77180b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.383+0000 7f771f563640 1 -- 172.21.15.52:0/3057814190 shutdown_connections 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.383+0000 7f771f563640 1 -- 172.21.15.52:0/3057814190 wait complete. 2024-03-25T19:30:35.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:35.383+0000 7f771f563640 1 librados: shutdown 2024-03-25T19:30:35.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:42.200 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:42.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.313+0000 7fba51f4d640 10 monclient: get_monmap_and_config 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.313+0000 7fba51f4d640 10 monclient: build_initial_monmap 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.313+0000 7fba51f4d640 10 monclient: monmap: 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:30:42.315004+0000 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:30:42.315004+0000 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:30:42.314 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.314+0000 7fba51f4d640 1 Processor -- start 2024-03-25T19:30:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.314+0000 7fba51f4d640 1 -- start start 2024-03-25T19:30:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.314+0000 7fba51f4d640 10 monclient: init 2024-03-25T19:30:42.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.314+0000 7fba51f4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.314+0000 7fba51f4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.314+0000 7fba51f4d640 10 monclient(hunting): picked mon.noname-a con 0x7fba4c0a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.315+0000 7fba51f4d640 10 monclient(hunting): picked mon.noname-c con 0x7fba4c13f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.315+0000 7fba51f4d640 10 monclient(hunting): picked mon.noname-b con 0x7fba4c1430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.315+0000 7fba51f4d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4c142520 con 0x7fba4c0a98f0 2024-03-25T19:30:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.315+0000 7fba51f4d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4c0b28a0 con 0x7fba4c13f710 2024-03-25T19:30:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.315+0000 7fba51f4d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4c0b2a30 con 0x7fba4c1430c0 2024-03-25T19:30:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.315+0000 7fba51f4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:42.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.315+0000 7fba51f4d640 10 monclient(hunting): authenticate will time out at 6090.495990s 2024-03-25T19:30:42.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.323+0000 7fba4b7fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fba4c0a98f0 0x7fba4c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:55824/0 (socket says 172.21.15.52:55824) 2024-03-25T19:30:42.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.323+0000 7fba4b7fe640 1 -- 172.21.15.52:0/1877729320 learned_addr learned my addr 172.21.15.52:0/1877729320 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:42.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.341+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331037664 0 0) 0x7fba4c0b2a30 con 0x7fba4c1430c0 2024-03-25T19:30:42.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.342+0000 7fba4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.342+0000 7fba4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.342+0000 7fba4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.342+0000 7fba4a7fc640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:30:42.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.342+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba34003520 con 0x7fba4c1430c0 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.345+0000 7fba4bfff640 1 -- 172.21.15.52:0/1877729320 >> v1:172.21.15.52:6790/0 conn(0x7fba4c1430c0 legacy=0x7fba4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.345+0000 7fba4bfff640 1 -- 172.21.15.52:0/1877729320 >> v1:172.21.15.52:6790/0 conn(0x7fba4c1430c0 legacy=0x7fba4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.345+0000 7fba4bfff640 1 --1- 172.21.15.52:0/1877729320 >> v1:172.21.15.52:6790/0 conn(0x7fba4c1430c0 0x7fba4c14dc30 :-1 s=OPENED pgs=20269 cs=1 l=1).handle_message read tag failed 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.345+0000 7fba4bfff640 1 --1- 172.21.15.52:0/1877729320 >> v1:172.21.15.52:6790/0 conn(0x7fba4c1430c0 0x7fba4c14dc30 :-1 s=OPENED pgs=20269 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:30:42.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.345+0000 7fba4a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.52:6790/0 2024-03-25T19:30:42.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.350+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201530326 0 0) 0x7fba4c142520 con 0x7fba4c0a98f0 2024-03-25T19:30:42.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.350+0000 7fba4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.350+0000 7fba4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.350+0000 7fba4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.350+0000 7fba4a7fc640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:30:42.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.350+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba4c0b2a30 con 0x7fba4c0a98f0 2024-03-25T19:30:42.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.354+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880284327 0 0) 0x7fba4c0b28a0 con 0x7fba4c13f710 2024-03-25T19:30:42.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.354+0000 7fba4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.354+0000 7fba4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.354+0000 7fba4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.354+0000 7fba4a7fc640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:30:42.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.354+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba4c142520 con 0x7fba4c13f710 2024-03-25T19:30:42.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1863452554 0 0) 0x7fba4c0b2a30 con 0x7fba4c0a98f0 2024-03-25T19:30:42.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 >> v1:172.21.15.52:6790/0 conn(0x7fba4c1430c0 legacy=0x7fba4c14dc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:30:42.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 >> v1:172.21.15.179:6789/0 conn(0x7fba4c13f710 legacy=0x7fba4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c0b44c0 con 0x7fba4c0a98f0 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fba3c001630 con 0x7fba4c0a98f0 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba51f4d640 5 monclient: authenticate success, global_id 4362 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba51f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba4a7fc640 10 monclient: dump: 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:42.363 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:42.364 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:42.364 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.364 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:42.364 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:42.364 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:42.364 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:42.364 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:42.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.362+0000 7fba51f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.390+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba3c001790 con 0x7fba4c0a98f0 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.390+0000 7fba4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.390+0000 7fba4a7fc640 1 -- 172.21.15.52:0/1877729320 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fba3c001ba0 con 0x7fba4c0a98f0 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.390+0000 7fba51f4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.390+0000 7fba51f4d640 4 set_mon_vals no callback set 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.390+0000 7fba4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.390+0000 7fba4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.390+0000 7fba4a7fc640 10 monclient: dump: 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:42.391 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:42.392 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.392 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:42.392 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:42.392 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:42.392 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:42.392 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.391+0000 7fba51f4d640 10 monclient: shutdown 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.391+0000 7fba51f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.391+0000 7fba51f4d640 1 -- 172.21.15.52:0/1877729320 >> v1:172.21.15.52:6789/0 conn(0x7fba4c0a98f0 legacy=0x7fba4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.391+0000 7fba51f4d640 1 -- 172.21.15.52:0/1877729320 shutdown_connections 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.391+0000 7fba51f4d640 1 -- 172.21.15.52:0/1877729320 >> 172.21.15.52:0/1877729320 conn(0x7fba4c0b7ea0 msgr2=0x7fba4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.391+0000 7fba51f4d640 1 -- 172.21.15.52:0/1877729320 shutdown_connections 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.391+0000 7fba51f4d640 1 -- 172.21.15.52:0/1877729320 wait complete. 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.392+0000 7fba51f4d640 10 monclient: build_initial_monmap 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.392+0000 7fba51f4d640 10 monclient: monmap: 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:42.394 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.392+0000 7fba51f4d640 1 librados: starting msgr at 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.392+0000 7fba51f4d640 1 librados: starting objecter 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.392+0000 7fba51f4d640 1 Processor -- start 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.392+0000 7fba51f4d640 1 -- start start 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.392+0000 7fba51f4d640 1 librados: setting wanted keys 2024-03-25T19:30:42.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.392+0000 7fba51f4d640 1 librados: calling monclient init 2024-03-25T19:30:42.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.392+0000 7fba51f4d640 10 monclient: init 2024-03-25T19:30:42.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.393+0000 7fba51f4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:42.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.393+0000 7fba51f4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:42.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.393+0000 7fba51f4d640 10 monclient(hunting): picked mon.noname-c con 0x7fba4c0a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:42.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.393+0000 7fba51f4d640 10 monclient(hunting): picked mon.noname-a con 0x7fba4c13f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:42.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.393+0000 7fba51f4d640 10 monclient(hunting): picked mon.noname-b con 0x7fba4c1430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:42.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.393+0000 7fba51f4d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4c0abb70 con 0x7fba4c13f710 2024-03-25T19:30:42.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.393+0000 7fba51f4d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4c138da0 con 0x7fba4c1430c0 2024-03-25T19:30:42.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.393+0000 7fba51f4d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4c138880 con 0x7fba4c0a98f0 2024-03-25T19:30:42.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.393+0000 7fba51f4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:42.396 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.393+0000 7fba51f4d640 10 monclient(hunting): authenticate will time out at 6090.574517s 2024-03-25T19:30:42.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.394+0000 7fba4b7fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fba4c0a98f0 0x7fba4c1379e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:50488/0 (socket says 172.21.15.52:50488) 2024-03-25T19:30:42.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.394+0000 7fba4b7fe640 1 -- 172.21.15.52:0/1072841635 learned_addr learned my addr 172.21.15.52:0/1072841635 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:42.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.395+0000 7fba33fff640 10 client.?.objecter ms_handle_connect 0x7fba4c0a98f0 2024-03-25T19:30:42.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.395+0000 7fba33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:42.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.408+0000 7fba33fff640 10 client.?.objecter ms_handle_connect 0x7fba4c1430c0 2024-03-25T19:30:42.408 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.408+0000 7fba33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:42.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.415+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926524190 0 0) 0x7fba4c138da0 con 0x7fba4c1430c0 2024-03-25T19:30:42.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.415+0000 7fba33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.415+0000 7fba33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.415+0000 7fba33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.415+0000 7fba33fff640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:30:42.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.415+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba38004eb0 con 0x7fba4c1430c0 2024-03-25T19:30:42.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.416+0000 7fba33fff640 10 client.?.objecter ms_handle_connect 0x7fba4c13f710 2024-03-25T19:30:42.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.416+0000 7fba33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:42.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.418+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843736034 0 0) 0x7fba4c138880 con 0x7fba4c0a98f0 2024-03-25T19:30:42.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.418+0000 7fba33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:42.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.418+0000 7fba33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:42.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.418+0000 7fba33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:42.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.418+0000 7fba33fff640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:30:42.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.418+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba4c138da0 con 0x7fba4c0a98f0 2024-03-25T19:30:42.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.427+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1968320842 0 0) 0x7fba38004eb0 con 0x7fba4c1430c0 2024-03-25T19:30:42.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.427+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba4c138880 con 0x7fba4c1430c0 2024-03-25T19:30:42.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.427+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fba44001c60 con 0x7fba4c1430c0 2024-03-25T19:30:42.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.427+0000 7fba33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.427+0000 7fba33fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.427+0000 7fba33fff640 10 monclient(hunting): dump: 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:42.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.427+0000 7fba33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:42.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1415231302 0 0) 0x7fba4c138da0 con 0x7fba4c0a98f0 2024-03-25T19:30:42.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba38004eb0 con 0x7fba4c0a98f0 2024-03-25T19:30:42.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fba3c001b60 con 0x7fba4c0a98f0 2024-03-25T19:30:42.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:42.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 10 monclient(hunting): dump: 2024-03-25T19:30:42.438 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:42.438 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 779083698 0 0) 0x7fba4c138880 con 0x7fba4c1430c0 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 >> v1:172.21.15.52:6790/0 conn(0x7fba4c0a98f0 legacy=0x7fba4c1379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 >> v1:172.21.15.52:6789/0 conn(0x7fba4c13f710 legacy=0x7fba4c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:42.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 1 monclient: found mon.b 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c0b18b0 con 0x7fba4c1430c0 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 10 monclient: _finish_auth 0 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba51f4d640 5 monclient: authenticate success, global_id 4375 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba51f4d640 10 monclient: _renew_subs 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba51f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba51f4d640 1 -- 172.21.15.52:0/1072841635 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4c138310 con 0x7fba4c1430c0 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba51f4d640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba51f4d640 10 monclient: _renew_subs 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba51f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:42.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba51f4d640 1 -- 172.21.15.52:0/1072841635 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba4c1db170 con 0x7fba4c1430c0 2024-03-25T19:30:42.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.437+0000 7fba51f4d640 1 librados: init done 2024-03-25T19:30:42.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.438+0000 7fba51f4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:42.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.438+0000 7fba51f4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:42.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.438+0000 7fba51f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:42.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.438+0000 7fba51f4d640 1 -- 172.21.15.52:0/1072841635 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4c138a10 con 0x7fba4c1430c0 2024-03-25T19:30:42.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.724+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba440020f0 con 0x7fba4c1430c0 2024-03-25T19:30:42.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.724+0000 7fba33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:42.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.724+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fba44003b30 con 0x7fba4c1430c0 2024-03-25T19:30:42.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.724+0000 7fba33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:42.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.724+0000 7fba33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:42.725 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.724+0000 7fba33fff640 10 monclient: dump: 2024-03-25T19:30:42.725 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:42.726 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:42.726 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:42.726 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:42.726 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:42.726 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:42.726 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:42.726 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:42.726 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:42.726 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:42.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.724+0000 7fba49ffb640 4 set_mon_vals no callback set 2024-03-25T19:30:42.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.725+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fba44003d50 con 0x7fba4c1430c0 2024-03-25T19:30:42.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.725+0000 7fba4b7fe640 10 monclient: get_auth_request con 0x7fba3806de40 auth_method 0 2024-03-25T19:30:42.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.726+0000 7fba33fff640 10 client.4375.objecter ms_handle_connect 0x7fba3806de40 2024-03-25T19:30:42.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.728+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fba44054a70 con 0x7fba4c1430c0 2024-03-25T19:30:42.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.728+0000 7fba33fff640 10 client.4375.objecter ms_dispatch 0x7fba4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:42.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.728+0000 7fba33fff640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:42.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.728+0000 7fba33fff640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:42.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.728+0000 7fba33fff640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-25T19:30:42.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.730+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba440503b0 con 0x7fba4c1430c0 2024-03-25T19:30:42.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.730+0000 7fba33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:42.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.730+0000 7fba33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:42.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.953+0000 7fba51f4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:42.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.953+0000 7fba51f4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:42.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.953+0000 7fba51f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:42.954 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:42.953+0000 7fba51f4d640 1 -- 172.21.15.52:0/1072841635 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba4c1386b0 con 0x7fba4c1430c0 2024-03-25T19:30:43.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.079+0000 7fba33fff640 1 -- 172.21.15.52:0/1072841635 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fba440524b0 con 0x7fba4c1430c0 2024-03-25T19:30:43.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.079+0000 7fba33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:43.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.079+0000 7fba33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:43.080 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:30:43.080 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:43.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 10 librados: watch_flush enter 2024-03-25T19:30:43.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 10 librados: watch_flush exit 2024-03-25T19:30:43.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-25T19:30:43.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 10 client.4375.objecter successfully canceled tick 2024-03-25T19:30:43.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 1 -- 172.21.15.52:0/1072841635 >> v1:172.21.15.52:6800/3967427147 conn(0x7fba3806de40 legacy=0x7fba38070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:43.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 10 monclient: shutdown 2024-03-25T19:30:43.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:43.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 1 -- 172.21.15.52:0/1072841635 >> v1:172.21.15.179:6789/0 conn(0x7fba4c1430c0 legacy=0x7fba4c1e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:43.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 1 -- 172.21.15.52:0/1072841635 shutdown_connections 2024-03-25T19:30:43.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 1 -- 172.21.15.52:0/1072841635 >> 172.21.15.52:0/1072841635 conn(0x7fba4c0b7ea0 msgr2=0x7fba4c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:43.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 1 -- 172.21.15.52:0/1072841635 shutdown_connections 2024-03-25T19:30:43.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 1 -- 172.21.15.52:0/1072841635 wait complete. 2024-03-25T19:30:43.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:43.082+0000 7fba51f4d640 1 librados: shutdown 2024-03-25T19:30:43.095 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:46.347 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:03:39.945 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:46.355 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:03:39.954 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:46.395 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:03:39.994 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:46.406 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:03:40.005 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:46.408 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:03:39.950 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:46.416 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:03:39.958 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:46.423 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:03:40.021 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:46.434 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:03:39.976 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:46.449 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:03:40.048 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:49.902 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:50.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.012+0000 7f46ec884640 10 monclient: get_monmap_and_config 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.012+0000 7f46ec884640 10 monclient: build_initial_monmap 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.012+0000 7f46ec884640 10 monclient: monmap: 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:30:50.014412+0000 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:30:50.014412+0000 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:30:50.014 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:50.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.013+0000 7f46ec884640 1 Processor -- start 2024-03-25T19:30:50.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.013+0000 7f46ec884640 1 -- start start 2024-03-25T19:30:50.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.013+0000 7f46ec884640 10 monclient: init 2024-03-25T19:30:50.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.014+0000 7f46ec884640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:50.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.014+0000 7f46ec884640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:50.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.014+0000 7f46ec884640 10 monclient(hunting): picked mon.noname-b con 0x7f46e80a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:50.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.014+0000 7f46ec884640 10 monclient(hunting): picked mon.noname-a con 0x7f46e813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:50.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.014+0000 7f46ec884640 10 monclient(hunting): picked mon.noname-c con 0x7f46e81430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:50.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.014+0000 7f46ec884640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e8142520 con 0x7f46e813f710 2024-03-25T19:30:50.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.014+0000 7f46ec884640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e80b28a0 con 0x7f46e81430c0 2024-03-25T19:30:50.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.014+0000 7f46ec884640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e80b2a30 con 0x7f46e80a98f0 2024-03-25T19:30:50.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.014+0000 7f46ec884640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:50.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.014+0000 7f46ec884640 10 monclient(hunting): authenticate will time out at 6098.195800s 2024-03-25T19:30:50.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.031+0000 7f46e659c640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f46e80a98f0 0x7f46e80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37992/0 (socket says 172.21.15.52:37992) 2024-03-25T19:30:50.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.031+0000 7f46e659c640 1 -- 172.21.15.52:0/3633394236 learned_addr learned my addr 172.21.15.52:0/3633394236 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:50.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.042+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352741974 0 0) 0x7f46e80b2a30 con 0x7f46e80a98f0 2024-03-25T19:30:50.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.042+0000 7f46e559a640 10 monclient(hunting): _negotiate 2024-03-25T19:30:50.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.042+0000 7f46e559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:50.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.042+0000 7f46e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:50.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.042+0000 7f46e559a640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:30:50.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.042+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d0003520 con 0x7f46e80a98f0 2024-03-25T19:30:50.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.050+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083426691 0 0) 0x7f46e80b28a0 con 0x7f46e81430c0 2024-03-25T19:30:50.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.050+0000 7f46e559a640 10 monclient(hunting): _negotiate 2024-03-25T19:30:50.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.050+0000 7f46e559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:50.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.050+0000 7f46e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:50.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.050+0000 7f46e559a640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:30:50.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.050+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e80b2a30 con 0x7f46e81430c0 2024-03-25T19:30:50.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.057+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75297660 0 0) 0x7f46e8142520 con 0x7f46e813f710 2024-03-25T19:30:50.057 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.057+0000 7f46e559a640 10 monclient(hunting): _negotiate 2024-03-25T19:30:50.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.057+0000 7f46e559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:50.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.057+0000 7f46e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:50.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.057+0000 7f46e559a640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:30:50.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.057+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e80b28a0 con 0x7f46e813f710 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3266322165 0 0) 0x7f46d0003520 con 0x7f46e80a98f0 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 >> v1:172.21.15.179:6789/0 conn(0x7f46e81430c0 legacy=0x7f46e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 >> v1:172.21.15.52:6789/0 conn(0x7f46e813f710 legacy=0x7f46e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 10 monclient: _finish_hunting 0 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 1 monclient: found mon.noname-b 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e80b44c0 con 0x7f46e80a98f0 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 10 monclient: _finish_auth 0 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 10 monclient: _check_auth_tickets 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:50.061 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46d8001690 con 0x7f46e80a98f0 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46ec884640 5 monclient: authenticate success, global_id 4367 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46ec884640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46e559a640 10 monclient: dump: 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:50.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.060+0000 7f46ec884640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.086+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46d80017f0 con 0x7f46e80a98f0 2024-03-25T19:30:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.086+0000 7f46e559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.086+0000 7f46e559a640 1 -- 172.21.15.52:0/3633394236 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46d8001c00 con 0x7f46e80a98f0 2024-03-25T19:30:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.086+0000 7f46ec884640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.086+0000 7f46ec884640 4 set_mon_vals no callback set 2024-03-25T19:30:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.086+0000 7f46e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.086+0000 7f46e559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.086+0000 7f46e559a640 10 monclient: dump: 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.086+0000 7f46ec884640 10 monclient: shutdown 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.086+0000 7f46ec884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.086+0000 7f46ec884640 1 -- 172.21.15.52:0/3633394236 >> v1:172.21.15.52:6790/0 conn(0x7f46e80a98f0 legacy=0x7f46e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.087+0000 7f46ec884640 1 -- 172.21.15.52:0/3633394236 shutdown_connections 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.087+0000 7f46ec884640 1 -- 172.21.15.52:0/3633394236 >> 172.21.15.52:0/3633394236 conn(0x7f46e80b7ea0 msgr2=0x7f46e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:50.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.087+0000 7f46ec884640 1 -- 172.21.15.52:0/3633394236 shutdown_connections 2024-03-25T19:30:50.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.087+0000 7f46ec884640 1 -- 172.21.15.52:0/3633394236 wait complete. 2024-03-25T19:30:50.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.087+0000 7f46ec884640 10 monclient: build_initial_monmap 2024-03-25T19:30:50.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.087+0000 7f46ec884640 10 monclient: monmap: 2024-03-25T19:30:50.090 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:50.090 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:50.090 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:30:50.090 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:30:50.090 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:50.090 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:30:50.090 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:50.090 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:30:50.090 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 1 librados: starting msgr at 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 1 librados: starting objecter 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 1 Processor -- start 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 1 -- start start 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 1 librados: setting wanted keys 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 1 librados: calling monclient init 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 10 monclient: init 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 10 monclient(hunting): picked mon.noname-b con 0x7f46e80a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 10 monclient(hunting): picked mon.noname-c con 0x7f46e813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 10 monclient(hunting): picked mon.noname-a con 0x7f46e81430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e80abd30 con 0x7f46e81430c0 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e80abfa0 con 0x7f46e80a98f0 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e81d77a0 con 0x7f46e813f710 2024-03-25T19:30:50.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:50.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.088+0000 7f46ec884640 10 monclient(hunting): authenticate will time out at 6098.269900s 2024-03-25T19:30:50.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.100+0000 7f46e659c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f46e80a98f0 0x7f46e81d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:37844/0 (socket says 172.21.15.52:37844) 2024-03-25T19:30:50.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.100+0000 7f46e659c640 1 -- 172.21.15.52:0/43544025 learned_addr learned my addr 172.21.15.52:0/43544025 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:50.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.108+0000 7f46d6ffd640 10 client.?.objecter ms_handle_connect 0x7f46e80a98f0 2024-03-25T19:30:50.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.108+0000 7f46d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:50.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.110+0000 7f46d6ffd640 10 client.?.objecter ms_handle_connect 0x7f46e81430c0 2024-03-25T19:30:50.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.110+0000 7f46d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:50.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.115+0000 7f46d6ffd640 10 client.?.objecter ms_handle_connect 0x7f46e813f710 2024-03-25T19:30:50.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.115+0000 7f46d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:50.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.122+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682585320 0 0) 0x7f46e80abfa0 con 0x7f46e80a98f0 2024-03-25T19:30:50.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.122+0000 7f46d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:50.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.122+0000 7f46d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:50.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.122+0000 7f46d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:50.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.122+0000 7f46d6ffd640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:30:50.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.122+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c8004eb0 con 0x7f46e80a98f0 2024-03-25T19:30:50.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.127+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974496556 0 0) 0x7f46e81d77a0 con 0x7f46e813f710 2024-03-25T19:30:50.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.127+0000 7f46d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:50.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.127+0000 7f46d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:50.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.127+0000 7f46d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:50.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.127+0000 7f46d6ffd640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:30:50.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.127+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e80abfa0 con 0x7f46e813f710 2024-03-25T19:30:50.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.129+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246857886 0 0) 0x7f46e80abd30 con 0x7f46e81430c0 2024-03-25T19:30:50.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.129+0000 7f46d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:50.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.129+0000 7f46d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:50.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.129+0000 7f46d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:50.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.129+0000 7f46d6ffd640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:30:50.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.129+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e81d77a0 con 0x7f46e81430c0 2024-03-25T19:30:50.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.133+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2192306770 0 0) 0x7f46c8004eb0 con 0x7f46e80a98f0 2024-03-25T19:30:50.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.133+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46e80abd30 con 0x7f46e80a98f0 2024-03-25T19:30:50.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.133+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46d8001bc0 con 0x7f46e80a98f0 2024-03-25T19:30:50.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.133+0000 7f46d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.133+0000 7f46d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:50.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.133+0000 7f46d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:50.134 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:50.134 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:50.135 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.135 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.135 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.135 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:50.135 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:50.135 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:50.135 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:50.135 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:50.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.133+0000 7f46d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:50.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.155+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1315084389 0 0) 0x7f46e80abfa0 con 0x7f46e813f710 2024-03-25T19:30:50.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.156+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46c8004eb0 con 0x7f46e813f710 2024-03-25T19:30:50.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.156+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46dc001870 con 0x7f46e813f710 2024-03-25T19:30:50.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.156+0000 7f46d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.156+0000 7f46d6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:50.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.156+0000 7f46d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:50.156 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:50.156 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.157+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2989823040 0 0) 0x7f46e81d77a0 con 0x7f46e81430c0 2024-03-25T19:30:50.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.157+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46e80abfa0 con 0x7f46e81430c0 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.159+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46e0001bd0 con 0x7f46e81430c0 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.159+0000 7f46d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.159+0000 7f46d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.159+0000 7f46d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:50.160 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:50.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2806266593 0 0) 0x7f46e80abd30 con 0x7f46e80a98f0 2024-03-25T19:30:50.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 >> v1:172.21.15.52:6790/0 conn(0x7f46e813f710 legacy=0x7f46e81d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 >> v1:172.21.15.52:6789/0 conn(0x7f46e81430c0 legacy=0x7f46e81385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:50.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46d6ffd640 1 monclient: found mon.b 2024-03-25T19:30:50.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:50.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46d6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:50.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e81d83d0 con 0x7f46e80a98f0 2024-03-25T19:30:50.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:50.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:50.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:50.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46ec884640 5 monclient: authenticate success, global_id 4381 2024-03-25T19:30:50.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46ec884640 10 monclient: _renew_subs 2024-03-25T19:30:50.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46ec884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:50.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46ec884640 1 -- 172.21.15.52:0/43544025 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e81d41e0 con 0x7f46e80a98f0 2024-03-25T19:30:50.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46ec884640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:50.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46ec884640 10 monclient: _renew_subs 2024-03-25T19:30:50.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46ec884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:50.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46ec884640 1 -- 172.21.15.52:0/43544025 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e81d5110 con 0x7f46e80a98f0 2024-03-25T19:30:50.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.162+0000 7f46ec884640 1 librados: init done 2024-03-25T19:30:50.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.163+0000 7f46ec884640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:50.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.163+0000 7f46ec884640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:50.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.163+0000 7f46ec884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:50.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.163+0000 7f46ec884640 1 -- 172.21.15.52:0/43544025 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e80627b0 con 0x7f46e80a98f0 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.442+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46d80011f0 con 0x7f46e80a98f0 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.442+0000 7f46d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.442+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46d8004cb0 con 0x7f46e80a98f0 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.442+0000 7f46d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.442+0000 7f46d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.442+0000 7f46d6ffd640 10 monclient: dump: 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:50.443 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:50.444 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:50.444 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:50.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.442+0000 7f46e4d99640 4 set_mon_vals no callback set 2024-03-25T19:30:50.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.443+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f46d801ad00 con 0x7f46e80a98f0 2024-03-25T19:30:50.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.444+0000 7f46e5d9b640 10 monclient: get_auth_request con 0x7f46c806dff0 auth_method 0 2024-03-25T19:30:50.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.444+0000 7f46d6ffd640 10 client.4381.objecter ms_handle_connect 0x7f46c806dff0 2024-03-25T19:30:50.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.449+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f46d8054950 con 0x7f46e80a98f0 2024-03-25T19:30:50.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.449+0000 7f46d6ffd640 10 client.4381.objecter ms_dispatch 0x7f46e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:50.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.449+0000 7f46d6ffd640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:50.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.449+0000 7f46d6ffd640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:50.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.449+0000 7f46d6ffd640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T19:30:50.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.449+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46d8050290 con 0x7f46e80a98f0 2024-03-25T19:30:50.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.449+0000 7f46d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:50.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.449+0000 7f46d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:50.706 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.706+0000 7f46ec884640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:50.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.706+0000 7f46ec884640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:50.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.706+0000 7f46ec884640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:30:50.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.706+0000 7f46ec884640 1 -- 172.21.15.52:0/43544025 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46e81d4370 con 0x7f46e80a98f0 2024-03-25T19:30:50.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.822+0000 7f46d6ffd640 1 -- 172.21.15.52:0/43544025 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f46d8052390 con 0x7f46e80a98f0 2024-03-25T19:30:50.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.822+0000 7f46d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:50.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.822+0000 7f46d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:50.823 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:30:50.823 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 10 librados: watch_flush enter 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 10 librados: watch_flush exit 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 10 client.4381.objecter successfully canceled tick 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 1 -- 172.21.15.52:0/43544025 >> v1:172.21.15.52:6800/3967427147 conn(0x7f46c806dff0 legacy=0x7f46c80704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 10 monclient: shutdown 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 1 -- 172.21.15.52:0/43544025 >> v1:172.21.15.179:6789/0 conn(0x7f46e80a98f0 legacy=0x7f46e81d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 1 -- 172.21.15.52:0/43544025 shutdown_connections 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 1 -- 172.21.15.52:0/43544025 >> 172.21.15.52:0/43544025 conn(0x7f46e80b7ea0 msgr2=0x7f46e813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 1 -- 172.21.15.52:0/43544025 shutdown_connections 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 1 -- 172.21.15.52:0/43544025 wait complete. 2024-03-25T19:30:50.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:50.825+0000 7f46ec884640 1 librados: shutdown 2024-03-25T19:30:50.838 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:57.644 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:57.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.757+0000 7f7e75693640 10 monclient: get_monmap_and_config 2024-03-25T19:30:57.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.757+0000 7f7e75693640 10 monclient: build_initial_monmap 2024-03-25T19:30:57.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.757+0000 7f7e75693640 10 monclient: monmap: 2024-03-25T19:30:57.758 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:57.758 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:57.758 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:30:57.759248+0000 2024-03-25T19:30:57.758 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:30:57.759248+0000 2024-03-25T19:30:57.758 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:57.759 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:57.759 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:57.759 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:30:57.759 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:30:57.759 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.758+0000 7f7e75693640 1 Processor -- start 2024-03-25T19:30:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.758+0000 7f7e75693640 1 -- start start 2024-03-25T19:30:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.758+0000 7f7e75693640 10 monclient: init 2024-03-25T19:30:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.759+0000 7f7e75693640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.759+0000 7f7e75693640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.759+0000 7f7e75693640 10 monclient(hunting): picked mon.noname-a con 0x7f7e700a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.759+0000 7f7e75693640 10 monclient(hunting): picked mon.noname-b con 0x7f7e7013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:57.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.759+0000 7f7e75693640 10 monclient(hunting): picked mon.noname-c con 0x7f7e701430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:57.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.759+0000 7f7e75693640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e70142520 con 0x7f7e700a98f0 2024-03-25T19:30:57.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.759+0000 7f7e75693640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e700b28a0 con 0x7f7e701430c0 2024-03-25T19:30:57.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.759+0000 7f7e75693640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e700b2a30 con 0x7f7e7013f710 2024-03-25T19:30:57.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.759+0000 7f7e75693640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:57.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.759+0000 7f7e75693640 10 monclient(hunting): authenticate will time out at 6105.940436s 2024-03-25T19:30:57.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.764+0000 7f7e67fff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f7e7013f710 0x7f7e701429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:34494/0 (socket says 172.21.15.52:34494) 2024-03-25T19:30:57.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.764+0000 7f7e67fff640 1 -- 172.21.15.52:0/1222380173 learned_addr learned my addr 172.21.15.52:0/1222380173 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:57.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.782+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570547893 0 0) 0x7f7e700b2a30 con 0x7f7e7013f710 2024-03-25T19:30:57.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.782+0000 7f7e6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.782+0000 7f7e6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.782+0000 7f7e6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.782+0000 7f7e6e7fc640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:30:57.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.782+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e5c003520 con 0x7f7e7013f710 2024-03-25T19:30:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.787+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195976697 0 0) 0x7f7e700b28a0 con 0x7f7e701430c0 2024-03-25T19:30:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.787+0000 7f7e6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.787+0000 7f7e6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.787+0000 7f7e6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.787+0000 7f7e6e7fc640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:30:57.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.787+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e700b2a30 con 0x7f7e701430c0 2024-03-25T19:30:57.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.795+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094874703 0 0) 0x7f7e70142520 con 0x7f7e700a98f0 2024-03-25T19:30:57.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.795+0000 7f7e6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.795+0000 7f7e6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.795+0000 7f7e6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.795+0000 7f7e6e7fc640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:30:57.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.795+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e700b28a0 con 0x7f7e700a98f0 2024-03-25T19:30:57.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4111541975 0 0) 0x7f7e700b2a30 con 0x7f7e701430c0 2024-03-25T19:30:57.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 >> v1:172.21.15.52:6790/0 conn(0x7f7e7013f710 legacy=0x7f7e701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 >> v1:172.21.15.52:6789/0 conn(0x7f7e700a98f0 legacy=0x7f7e700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:57.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:30:57.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:57.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:30:57.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e700b44c0 con 0x7f7e701430c0 2024-03-25T19:30:57.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:57.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:57.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7e680019f0 con 0x7f7e701430c0 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e6e7fc640 10 monclient: dump: 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e75693640 5 monclient: authenticate success, global_id 4384 2024-03-25T19:30:57.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.807+0000 7f7e75693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:57.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.815+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e68001b50 con 0x7f7e701430c0 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.815+0000 7f7e6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.815+0000 7f7e6e7fc640 1 -- 172.21.15.52:0/1222380173 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7e68001f60 con 0x7f7e701430c0 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.815+0000 7f7e6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.815+0000 7f7e6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.815+0000 7f7e6e7fc640 10 monclient: dump: 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.815+0000 7f7e75693640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:57.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.815+0000 7f7e75693640 4 set_mon_vals no callback set 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.816+0000 7f7e75693640 10 monclient: shutdown 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.816+0000 7f7e75693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.816+0000 7f7e75693640 1 -- 172.21.15.52:0/1222380173 >> v1:172.21.15.179:6789/0 conn(0x7f7e701430c0 legacy=0x7f7e7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.816+0000 7f7e75693640 1 -- 172.21.15.52:0/1222380173 shutdown_connections 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.816+0000 7f7e75693640 1 -- 172.21.15.52:0/1222380173 >> 172.21.15.52:0/1222380173 conn(0x7f7e700b7ea0 msgr2=0x7f7e700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.816+0000 7f7e75693640 1 -- 172.21.15.52:0/1222380173 shutdown_connections 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.816+0000 7f7e75693640 1 -- 172.21.15.52:0/1222380173 wait complete. 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.817+0000 7f7e75693640 10 monclient: build_initial_monmap 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.817+0000 7f7e75693640 10 monclient: monmap: 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:57.818 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.817+0000 7f7e75693640 1 librados: starting msgr at 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.817+0000 7f7e75693640 1 librados: starting objecter 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.817+0000 7f7e75693640 1 Processor -- start 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.817+0000 7f7e75693640 1 -- start start 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.817+0000 7f7e75693640 1 librados: setting wanted keys 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.817+0000 7f7e75693640 1 librados: calling monclient init 2024-03-25T19:30:57.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.817+0000 7f7e75693640 10 monclient: init 2024-03-25T19:30:57.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.818+0000 7f7e75693640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:57.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.818+0000 7f7e75693640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:57.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.818+0000 7f7e75693640 10 monclient(hunting): picked mon.noname-b con 0x7f7e700a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:30:57.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.818+0000 7f7e75693640 10 monclient(hunting): picked mon.noname-c con 0x7f7e7013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:30:57.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.818+0000 7f7e75693640 10 monclient(hunting): picked mon.noname-a con 0x7f7e701430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:30:57.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.818+0000 7f7e75693640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e701daaf0 con 0x7f7e701430c0 2024-03-25T19:30:57.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.818+0000 7f7e75693640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e70136e50 con 0x7f7e700a98f0 2024-03-25T19:30:57.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.818+0000 7f7e75693640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e70137130 con 0x7f7e7013f710 2024-03-25T19:30:57.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.818+0000 7f7e75693640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:57.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.818+0000 7f7e75693640 10 monclient(hunting): authenticate will time out at 6105.999519s 2024-03-25T19:30:57.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.821+0000 7f7e6f7fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f7e701430c0 0x7f7e701e2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:50528/0 (socket says 172.21.15.52:50528) 2024-03-25T19:30:57.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.821+0000 7f7e6f7fe640 1 -- 172.21.15.52:0/4146476369 learned_addr learned my addr 172.21.15.52:0/4146476369 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:30:57.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.826+0000 7f7e677fe640 10 client.?.objecter ms_handle_connect 0x7f7e701430c0 2024-03-25T19:30:57.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.826+0000 7f7e677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:57.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.829+0000 7f7e677fe640 10 client.?.objecter ms_handle_connect 0x7f7e7013f710 2024-03-25T19:30:57.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.829+0000 7f7e677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:57.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.831+0000 7f7e677fe640 10 client.?.objecter ms_handle_connect 0x7f7e700a98f0 2024-03-25T19:30:57.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.831+0000 7f7e677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.834+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002075970 0 0) 0x7f7e701daaf0 con 0x7f7e701430c0 2024-03-25T19:30:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.834+0000 7f7e677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.834+0000 7f7e677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.834+0000 7f7e677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.834+0000 7f7e677fe640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:30:57.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.834+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e50004eb0 con 0x7f7e701430c0 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.837+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673311644 0 0) 0x7f7e70136e50 con 0x7f7e700a98f0 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.837+0000 7f7e677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.837+0000 7f7e677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.837+0000 7f7e677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.837+0000 7f7e677fe640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:30:57.838 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.837+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e701daaf0 con 0x7f7e700a98f0 2024-03-25T19:30:57.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.846+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1589276301 0 0) 0x7f7e50004eb0 con 0x7f7e701430c0 2024-03-25T19:30:57.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.846+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e70136e50 con 0x7f7e701430c0 2024-03-25T19:30:57.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.846+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7e68001fa0 con 0x7f7e701430c0 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.846+0000 7f7e677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.846+0000 7f7e677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.846+0000 7f7e677fe640 10 monclient(hunting): dump: 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:57.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.846+0000 7f7e677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.852+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733655907 0 0) 0x7f7e70137130 con 0x7f7e7013f710 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.852+0000 7f7e677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.852+0000 7f7e677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.852+0000 7f7e677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:57.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.852+0000 7f7e677fe640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:30:57.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.852+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e50004eb0 con 0x7f7e7013f710 2024-03-25T19:30:57.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1304038654 0 0) 0x7f7e70136e50 con 0x7f7e701430c0 2024-03-25T19:30:57.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 >> v1:172.21.15.52:6790/0 conn(0x7f7e7013f710 legacy=0x7f7e70135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 >> v1:172.21.15.179:6789/0 conn(0x7f7e700a98f0 legacy=0x7f7e701daf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e677fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:57.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e677fe640 1 monclient: found mon.a 2024-03-25T19:30:57.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e677fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e70139f80 con 0x7f7e701430c0 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e677fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e677fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e75693640 5 monclient: authenticate success, global_id 4377 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e75693640 10 monclient: _renew_subs 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e75693640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e75693640 1 -- 172.21.15.52:0/4146476369 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e701db690 con 0x7f7e701430c0 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e75693640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e75693640 10 monclient: _renew_subs 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e75693640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e75693640 1 -- 172.21.15.52:0/4146476369 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e70135c60 con 0x7f7e701430c0 2024-03-25T19:30:57.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.871+0000 7f7e75693640 1 librados: init done 2024-03-25T19:30:57.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.872+0000 7f7e75693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:57.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.872+0000 7f7e75693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:57.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.872+0000 7f7e75693640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:30:57.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:57.872+0000 7f7e75693640 1 -- 172.21.15.52:0/4146476369 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e700b3ef0 con 0x7f7e701430c0 2024-03-25T19:30:58.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.200+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e68003c20 con 0x7f7e701430c0 2024-03-25T19:30:58.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.200+0000 7f7e677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:58.200 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.200+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7e68003f70 con 0x7f7e701430c0 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.200+0000 7f7e677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.200+0000 7f7e677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.200+0000 7f7e677fe640 10 monclient: dump: 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:30:58.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.200+0000 7f7e6dffb640 4 set_mon_vals no callback set 2024-03-25T19:30:58.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.200+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f7e68004190 con 0x7f7e701430c0 2024-03-25T19:30:58.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.201+0000 7f7e6effd640 10 monclient: get_auth_request con 0x7f7e5006de00 auth_method 0 2024-03-25T19:30:58.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.201+0000 7f7e677fe640 10 client.4377.objecter ms_handle_connect 0x7f7e5006de00 2024-03-25T19:30:58.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.206+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f7e68054bc0 con 0x7f7e701430c0 2024-03-25T19:30:58.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.206+0000 7f7e677fe640 10 client.4377.objecter ms_dispatch 0x7f7e70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:58.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.206+0000 7f7e677fe640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:58.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.206+0000 7f7e677fe640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:58.207 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.206+0000 7f7e677fe640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-25T19:30:58.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.206+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e680506d0 con 0x7f7e701430c0 2024-03-25T19:30:58.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.206+0000 7f7e677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:58.208 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.206+0000 7f7e677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:58.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.440+0000 7f7e75693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:58.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.440+0000 7f7e75693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:58.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.440+0000 7f7e75693640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:30:58.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.440+0000 7f7e75693640 1 -- 172.21.15.52:0/4146476369 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e701daca0 con 0x7f7e701430c0 2024-03-25T19:30:58.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.474+0000 7f7e677fe640 1 -- 172.21.15.52:0/4146476369 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f7e701daca0 con 0x7f7e701430c0 2024-03-25T19:30:58.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.474+0000 7f7e677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:58.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.474+0000 7f7e677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:58.474 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:30:58.475 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:58.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 10 librados: watch_flush enter 2024-03-25T19:30:58.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 10 librados: watch_flush exit 2024-03-25T19:30:58.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-25T19:30:58.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 10 client.4377.objecter successfully canceled tick 2024-03-25T19:30:58.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 1 -- 172.21.15.52:0/4146476369 >> v1:172.21.15.52:6800/3967427147 conn(0x7f7e5006de00 legacy=0x7f7e500702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:58.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 10 monclient: shutdown 2024-03-25T19:30:58.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:58.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 1 -- 172.21.15.52:0/4146476369 >> v1:172.21.15.52:6789/0 conn(0x7f7e701430c0 legacy=0x7f7e701e2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:58.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 1 -- 172.21.15.52:0/4146476369 shutdown_connections 2024-03-25T19:30:58.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 1 -- 172.21.15.52:0/4146476369 >> 172.21.15.52:0/4146476369 conn(0x7f7e700b7ea0 msgr2=0x7f7e70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:58.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 1 -- 172.21.15.52:0/4146476369 shutdown_connections 2024-03-25T19:30:58.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 1 -- 172.21.15.52:0/4146476369 wait complete. 2024-03-25T19:30:58.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:30:58.476+0000 7f7e75693640 1 librados: shutdown 2024-03-25T19:30:58.489 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:05.295 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:05.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.408+0000 7fa86d84a640 10 monclient: get_monmap_and_config 2024-03-25T19:31:05.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.408+0000 7fa86d84a640 10 monclient: build_initial_monmap 2024-03-25T19:31:05.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.408+0000 7fa86d84a640 10 monclient: monmap: 2024-03-25T19:31:05.409 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:05.409 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:05.409 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:31:05.410205+0000 2024-03-25T19:31:05.409 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:31:05.410205+0000 2024-03-25T19:31:05.409 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:05.409 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:05.410 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:05.410 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:31:05.410 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:31:05.410 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:05.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.409+0000 7fa86d84a640 1 Processor -- start 2024-03-25T19:31:05.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.409+0000 7fa86d84a640 1 -- start start 2024-03-25T19:31:05.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.409+0000 7fa86d84a640 10 monclient: init 2024-03-25T19:31:05.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.409+0000 7fa86d84a640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:05.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.409+0000 7fa86d84a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:05.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.409+0000 7fa86d84a640 10 monclient(hunting): picked mon.noname-b con 0x7fa8680a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:05.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.409+0000 7fa86d84a640 10 monclient(hunting): picked mon.noname-c con 0x7fa86813f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:05.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.409+0000 7fa86d84a640 10 monclient(hunting): picked mon.noname-a con 0x7fa8681430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:05.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.410+0000 7fa86d84a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa868142520 con 0x7fa8681430c0 2024-03-25T19:31:05.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.410+0000 7fa86d84a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8680b28a0 con 0x7fa86813f710 2024-03-25T19:31:05.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.410+0000 7fa86d84a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8680b2a30 con 0x7fa8680a98f0 2024-03-25T19:31:05.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.410+0000 7fa86d84a640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:05.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.410+0000 7fa86d84a640 10 monclient(hunting): authenticate will time out at 6113.591174s 2024-03-25T19:31:05.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.418+0000 7fa866ffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fa8680a98f0 0x7fa8680a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:35106/0 (socket says 172.21.15.52:35106) 2024-03-25T19:31:05.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.418+0000 7fa866ffd640 1 -- 172.21.15.52:0/227572248 learned_addr learned my addr 172.21.15.52:0/227572248 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:05.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.439+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026917213 0 0) 0x7fa8680b28a0 con 0x7fa86813f710 2024-03-25T19:31:05.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.439+0000 7fa865ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:05.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.439+0000 7fa865ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:05.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.439+0000 7fa865ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:05.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.439+0000 7fa865ffb640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:31:05.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.439+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa850003520 con 0x7fa86813f710 2024-03-25T19:31:05.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.442+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283014873 0 0) 0x7fa868142520 con 0x7fa8681430c0 2024-03-25T19:31:05.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.442+0000 7fa865ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:05.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.442+0000 7fa865ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:05.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.442+0000 7fa865ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:05.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.442+0000 7fa865ffb640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:31:05.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.442+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8680b28a0 con 0x7fa8681430c0 2024-03-25T19:31:05.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.445+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802193013 0 0) 0x7fa8680b2a30 con 0x7fa8680a98f0 2024-03-25T19:31:05.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.445+0000 7fa865ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:05.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.445+0000 7fa865ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:05.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.445+0000 7fa865ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:05.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.445+0000 7fa865ffb640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:31:05.446 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.445+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa868142520 con 0x7fa8680a98f0 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3768011296 0 0) 0x7fa850003520 con 0x7fa86813f710 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 >> v1:172.21.15.52:6790/0 conn(0x7fa8680a98f0 legacy=0x7fa8680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 >> v1:172.21.15.52:6789/0 conn(0x7fa8681430c0 legacy=0x7fa86814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 1 monclient: found mon.noname-c 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8680b44c0 con 0x7fa86813f710 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa85c001690 con 0x7fa86813f710 2024-03-25T19:31:05.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa86d84a640 5 monclient: authenticate success, global_id 4390 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa86d84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa865ffb640 10 monclient: dump: 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:05.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.450+0000 7fa86d84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:05.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.457+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa85c0017f0 con 0x7fa86813f710 2024-03-25T19:31:05.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.457+0000 7fa865ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:05.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.457+0000 7fa865ffb640 1 -- 172.21.15.52:0/227572248 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa85c001c00 con 0x7fa86813f710 2024-03-25T19:31:05.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.457+0000 7fa865ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:05.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.457+0000 7fa865ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:05.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.457+0000 7fa865ffb640 10 monclient: dump: 2024-03-25T19:31:05.458 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:05.458 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:05.458 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:05.458 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:05.458 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:05.459 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:05.459 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:05.459 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:05.459 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:05.459 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:05.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.457+0000 7fa86d84a640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:05.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.457+0000 7fa86d84a640 4 set_mon_vals no callback set 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.458+0000 7fa86d84a640 10 monclient: shutdown 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.458+0000 7fa86d84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.458+0000 7fa86d84a640 1 -- 172.21.15.52:0/227572248 >> v1:172.21.15.179:6789/0 conn(0x7fa86813f710 legacy=0x7fa8681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.458+0000 7fa86d84a640 1 -- 172.21.15.52:0/227572248 shutdown_connections 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.458+0000 7fa86d84a640 1 -- 172.21.15.52:0/227572248 >> 172.21.15.52:0/227572248 conn(0x7fa8680b7ea0 msgr2=0x7fa8680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.459+0000 7fa86d84a640 1 -- 172.21.15.52:0/227572248 shutdown_connections 2024-03-25T19:31:05.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.459+0000 7fa86d84a640 1 -- 172.21.15.52:0/227572248 wait complete. 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.460+0000 7fa86d84a640 10 monclient: build_initial_monmap 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.460+0000 7fa86d84a640 10 monclient: monmap: 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:31:05.461 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:31:05.462 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:05.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.460+0000 7fa86d84a640 1 librados: starting msgr at 2024-03-25T19:31:05.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.460+0000 7fa86d84a640 1 librados: starting objecter 2024-03-25T19:31:05.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.460+0000 7fa86d84a640 1 Processor -- start 2024-03-25T19:31:05.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.460+0000 7fa86d84a640 1 -- start start 2024-03-25T19:31:05.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.460+0000 7fa86d84a640 1 librados: setting wanted keys 2024-03-25T19:31:05.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.460+0000 7fa86d84a640 1 librados: calling monclient init 2024-03-25T19:31:05.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.460+0000 7fa86d84a640 10 monclient: init 2024-03-25T19:31:05.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.461+0000 7fa86d84a640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:05.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.461+0000 7fa86d84a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:05.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.461+0000 7fa86d84a640 10 monclient(hunting): picked mon.noname-c con 0x7fa8680a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:05.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.461+0000 7fa86d84a640 10 monclient(hunting): picked mon.noname-a con 0x7fa86813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:05.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.461+0000 7fa86d84a640 10 monclient(hunting): picked mon.noname-b con 0x7fa8681430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:05.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.461+0000 7fa86d84a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8681d77a0 con 0x7fa86813f710 2024-03-25T19:31:05.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.461+0000 7fa86d84a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa868138e40 con 0x7fa8681430c0 2024-03-25T19:31:05.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.461+0000 7fa86d84a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8680aba30 con 0x7fa8680a98f0 2024-03-25T19:31:05.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.461+0000 7fa86d84a640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:05.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.461+0000 7fa86d84a640 10 monclient(hunting): authenticate will time out at 6113.642413s 2024-03-25T19:31:05.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.463+0000 7fa8667fc640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fa86813f710 0x7fa8681d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:58944/0 (socket says 172.21.15.52:58944) 2024-03-25T19:31:05.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.463+0000 7fa8667fc640 1 -- 172.21.15.52:0/3330671290 learned_addr learned my addr 172.21.15.52:0/3330671290 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:05.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.463+0000 7fa84ffff640 10 client.?.objecter ms_handle_connect 0x7fa86813f710 2024-03-25T19:31:05.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.463+0000 7fa84ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:05.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.472+0000 7fa84ffff640 10 client.?.objecter ms_handle_connect 0x7fa8681430c0 2024-03-25T19:31:05.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.472+0000 7fa84ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:05.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.473+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998962136 0 0) 0x7fa8681d77a0 con 0x7fa86813f710 2024-03-25T19:31:05.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.473+0000 7fa84ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:05.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.473+0000 7fa84ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:05.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.473+0000 7fa84ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:05.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.473+0000 7fa84ffff640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:31:05.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.473+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa854004eb0 con 0x7fa86813f710 2024-03-25T19:31:05.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.478+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896072972 0 0) 0x7fa868138e40 con 0x7fa8681430c0 2024-03-25T19:31:05.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.478+0000 7fa84ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:05.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.478+0000 7fa84ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:05.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.478+0000 7fa84ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.478+0000 7fa84ffff640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:31:05.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.478+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8681d77a0 con 0x7fa8681430c0 2024-03-25T19:31:05.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.480+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 963336226 0 0) 0x7fa854004eb0 con 0x7fa86813f710 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.480+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa868138e40 con 0x7fa86813f710 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.480+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa85c001bc0 con 0x7fa86813f710 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.480+0000 7fa84ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.480+0000 7fa84ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.480+0000 7fa84ffff640 10 monclient(hunting): dump: 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:05.481 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:05.482 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:05.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.480+0000 7fa84ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:05.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.483+0000 7fa84ffff640 10 client.?.objecter ms_handle_connect 0x7fa8680a98f0 2024-03-25T19:31:05.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.483+0000 7fa84ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2739885023 0 0) 0x7fa868138e40 con 0x7fa86813f710 2024-03-25T19:31:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 >> v1:172.21.15.52:6790/0 conn(0x7fa8680a98f0 legacy=0x7fa868138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 >> v1:172.21.15.179:6789/0 conn(0x7fa8681430c0 legacy=0x7fa868134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa84ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa84ffff640 1 monclient: found mon.a 2024-03-25T19:31:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa84ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa84ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8681d8340 con 0x7fa86813f710 2024-03-25T19:31:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa84ffff640 10 monclient: _finish_auth 0 2024-03-25T19:31:05.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa84ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa84ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa86d84a640 5 monclient: authenticate success, global_id 4383 2024-03-25T19:31:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa86d84a640 10 monclient: _renew_subs 2024-03-25T19:31:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa86d84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa86d84a640 1 -- 172.21.15.52:0/3330671290 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa868134820 con 0x7fa86813f710 2024-03-25T19:31:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa86d84a640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa86d84a640 10 monclient: _renew_subs 2024-03-25T19:31:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa86d84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa86d84a640 1 -- 172.21.15.52:0/3330671290 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8681380c0 con 0x7fa86813f710 2024-03-25T19:31:05.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.490+0000 7fa86d84a640 1 librados: init done 2024-03-25T19:31:05.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.491+0000 7fa86d84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:05.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.491+0000 7fa86d84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:05.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.491+0000 7fa86d84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:05.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.491+0000 7fa86d84a640 1 -- 172.21.15.52:0/3330671290 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8680627b0 con 0x7fa86813f710 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.800+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa85c0011f0 con 0x7fa86813f710 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.800+0000 7fa84ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.800+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa85c003b00 con 0x7fa86813f710 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.800+0000 7fa84ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.800+0000 7fa84ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.800+0000 7fa84ffff640 10 monclient: dump: 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:05.801 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:05.802 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:05.802 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:05.802 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:05.802 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:05.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.800+0000 7fa8657fa640 4 set_mon_vals no callback set 2024-03-25T19:31:05.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.800+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fa85c003d20 con 0x7fa86813f710 2024-03-25T19:31:05.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.801+0000 7fa866ffd640 10 monclient: get_auth_request con 0x7fa85406dd40 auth_method 0 2024-03-25T19:31:05.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.801+0000 7fa84ffff640 10 client.4383.objecter ms_handle_connect 0x7fa85406dd40 2024-03-25T19:31:05.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.809+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fa85c054880 con 0x7fa86813f710 2024-03-25T19:31:05.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.809+0000 7fa84ffff640 10 client.4383.objecter ms_dispatch 0x7fa868000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:05.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.809+0000 7fa84ffff640 3 client.4383.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:05.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.809+0000 7fa84ffff640 3 client.4383.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:05.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.809+0000 7fa84ffff640 20 client.4383.objecter dump_active .. 0 homeless 2024-03-25T19:31:05.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.809+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa85c050390 con 0x7fa86813f710 2024-03-25T19:31:05.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.809+0000 7fa84ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:05.810 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:05.809+0000 7fa84ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:06.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.037+0000 7fa86d84a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:06.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.037+0000 7fa86d84a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:06.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.037+0000 7fa86d84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:06.037 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.037+0000 7fa86d84a640 1 -- 172.21.15.52:0/3330671290 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa86814e8b0 con 0x7fa86813f710 2024-03-25T19:31:06.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.055+0000 7fa84ffff640 1 -- 172.21.15.52:0/3330671290 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fa85c0522c0 con 0x7fa86813f710 2024-03-25T19:31:06.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.055+0000 7fa84ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:06.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.055+0000 7fa84ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:06.055 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:31:06.056 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:06.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.057+0000 7fa86d84a640 10 librados: watch_flush enter 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.057+0000 7fa86d84a640 10 librados: watch_flush exit 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.058+0000 7fa86d84a640 20 client.4383.objecter shutdown clearing up homeless session... 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.058+0000 7fa86d84a640 10 client.4383.objecter successfully canceled tick 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.058+0000 7fa86d84a640 1 -- 172.21.15.52:0/3330671290 >> v1:172.21.15.52:6800/3967427147 conn(0x7fa85406dd40 legacy=0x7fa854070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.058+0000 7fa86d84a640 10 monclient: shutdown 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.058+0000 7fa86d84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.058+0000 7fa86d84a640 1 -- 172.21.15.52:0/3330671290 >> v1:172.21.15.52:6789/0 conn(0x7fa86813f710 legacy=0x7fa8681d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.058+0000 7fa86d84a640 1 -- 172.21.15.52:0/3330671290 shutdown_connections 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.058+0000 7fa86d84a640 1 -- 172.21.15.52:0/3330671290 >> 172.21.15.52:0/3330671290 conn(0x7fa8680b7ea0 msgr2=0x7fa868074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.058+0000 7fa86d84a640 1 -- 172.21.15.52:0/3330671290 shutdown_connections 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.058+0000 7fa86d84a640 1 -- 172.21.15.52:0/3330671290 wait complete. 2024-03-25T19:31:06.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:06.058+0000 7fa86d84a640 1 librados: shutdown 2024-03-25T19:31:06.070 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:06.421 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:04:00.020 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:06.429 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:04:00.028 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:06.464 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:04:00.063 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:06.469 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:04:00.011 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:06.478 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:04:00.077 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:06.478 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:04:00.020 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:06.483 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:04:00.082 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:06.501 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:04:00.043 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:06.515 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:04:00.114 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.878 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:12.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.993+0000 7fb7597d7640 10 monclient: get_monmap_and_config 2024-03-25T19:31:12.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.993+0000 7fb7597d7640 10 monclient: build_initial_monmap 2024-03-25T19:31:12.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.993+0000 7fb7597d7640 10 monclient: monmap: 2024-03-25T19:31:12.994 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:12.994 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:12.994 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:31:12.994742+0000 2024-03-25T19:31:12.994 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:31:12.994742+0000 2024-03-25T19:31:12.994 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:12.994 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:12.994 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:12.994 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:31:12.994 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:31:12.994 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:12.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.993+0000 7fb7597d7640 1 Processor -- start 2024-03-25T19:31:12.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.993+0000 7fb7597d7640 1 -- start start 2024-03-25T19:31:12.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.993+0000 7fb7597d7640 10 monclient: init 2024-03-25T19:31:12.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.994+0000 7fb7597d7640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:12.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.994+0000 7fb7597d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:12.995 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.994+0000 7fb7597d7640 10 monclient(hunting): picked mon.noname-a con 0x7fb7540a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:12.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.994+0000 7fb7597d7640 10 monclient(hunting): picked mon.noname-c con 0x7fb75413f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:12.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.994+0000 7fb7597d7640 10 monclient(hunting): picked mon.noname-b con 0x7fb7541430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:12.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.994+0000 7fb7597d7640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb754142520 con 0x7fb7540a98f0 2024-03-25T19:31:12.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.994+0000 7fb7597d7640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7540b28a0 con 0x7fb75413f710 2024-03-25T19:31:12.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.994+0000 7fb7597d7640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7540b2a30 con 0x7fb7541430c0 2024-03-25T19:31:12.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.994+0000 7fb7597d7640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:12.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:12.994+0000 7fb7597d7640 10 monclient(hunting): authenticate will time out at 6121.175474s 2024-03-25T19:31:13.006 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.005+0000 7fb7527fc640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fb75413f710 0x7fb7541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:58724/0 (socket says 172.21.15.52:58724) 2024-03-25T19:31:13.006 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.005+0000 7fb7527fc640 1 -- 172.21.15.52:0/3892281757 learned_addr learned my addr 172.21.15.52:0/3892281757 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:13.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.025+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131863350 0 0) 0x7fb7540b2a30 con 0x7fb7541430c0 2024-03-25T19:31:13.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.025+0000 7fb751ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:13.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.025+0000 7fb751ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:13.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.025+0000 7fb751ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:13.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.025+0000 7fb751ffb640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:31:13.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.025+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb73c003520 con 0x7fb7541430c0 2024-03-25T19:31:13.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.030+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59608075 0 0) 0x7fb7540b28a0 con 0x7fb75413f710 2024-03-25T19:31:13.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.030+0000 7fb751ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:13.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.030+0000 7fb751ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:13.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.030+0000 7fb751ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:13.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.030+0000 7fb751ffb640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:31:13.031 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.030+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7540b2a30 con 0x7fb75413f710 2024-03-25T19:31:13.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.032+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140781398 0 0) 0x7fb754142520 con 0x7fb7540a98f0 2024-03-25T19:31:13.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.032+0000 7fb751ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:13.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.032+0000 7fb751ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:13.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.032+0000 7fb751ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:13.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.032+0000 7fb751ffb640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:31:13.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.032+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7540b28a0 con 0x7fb7540a98f0 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 971041099 0 0) 0x7fb7540b2a30 con 0x7fb75413f710 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 >> v1:172.21.15.52:6790/0 conn(0x7fb7541430c0 legacy=0x7fb75414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 >> v1:172.21.15.52:6789/0 conn(0x7fb7540a98f0 legacy=0x7fb7540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 1 monclient: found mon.noname-c 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7540b44c0 con 0x7fb75413f710 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:13.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb748001670 con 0x7fb75413f710 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb7597d7640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb7597d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb751ffb640 10 monclient: dump: 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:13.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.054+0000 7fb7597d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:13.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.064+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7480017d0 con 0x7fb75413f710 2024-03-25T19:31:13.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.064+0000 7fb751ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:13.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.064+0000 7fb751ffb640 1 -- 172.21.15.52:0/3892281757 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb748001be0 con 0x7fb75413f710 2024-03-25T19:31:13.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.064+0000 7fb751ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:13.064 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.064+0000 7fb751ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.064+0000 7fb751ffb640 10 monclient: dump: 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.064+0000 7fb7597d7640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:13.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.064+0000 7fb7597d7640 4 set_mon_vals no callback set 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.064+0000 7fb7597d7640 10 monclient: shutdown 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.064+0000 7fb7597d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.064+0000 7fb7597d7640 1 -- 172.21.15.52:0/3892281757 >> v1:172.21.15.179:6789/0 conn(0x7fb75413f710 legacy=0x7fb7541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.065+0000 7fb7597d7640 1 -- 172.21.15.52:0/3892281757 shutdown_connections 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.065+0000 7fb7597d7640 1 -- 172.21.15.52:0/3892281757 >> 172.21.15.52:0/3892281757 conn(0x7fb7540b7ea0 msgr2=0x7fb7540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.065+0000 7fb7597d7640 1 -- 172.21.15.52:0/3892281757 shutdown_connections 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.065+0000 7fb7597d7640 1 -- 172.21.15.52:0/3892281757 wait complete. 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.066+0000 7fb7597d7640 10 monclient: build_initial_monmap 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.066+0000 7fb7597d7640 10 monclient: monmap: 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:31:13.067 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.066+0000 7fb7597d7640 1 librados: starting msgr at 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.066+0000 7fb7597d7640 1 librados: starting objecter 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.066+0000 7fb7597d7640 1 Processor -- start 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.066+0000 7fb7597d7640 1 -- start start 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.066+0000 7fb7597d7640 1 librados: setting wanted keys 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.066+0000 7fb7597d7640 1 librados: calling monclient init 2024-03-25T19:31:13.068 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.066+0000 7fb7597d7640 10 monclient: init 2024-03-25T19:31:13.069 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.067+0000 7fb7597d7640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:13.069 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.067+0000 7fb7597d7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:13.069 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.067+0000 7fb7597d7640 10 monclient(hunting): picked mon.noname-b con 0x7fb7540a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:13.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.067+0000 7fb7597d7640 10 monclient(hunting): picked mon.noname-c con 0x7fb75413f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:13.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.067+0000 7fb7597d7640 10 monclient(hunting): picked mon.noname-a con 0x7fb7541430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:13.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.067+0000 7fb7597d7640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7541d7a90 con 0x7fb7541430c0 2024-03-25T19:31:13.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.067+0000 7fb7597d7640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7540b1700 con 0x7fb7540a98f0 2024-03-25T19:31:13.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.067+0000 7fb7597d7640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7540b13d0 con 0x7fb75413f710 2024-03-25T19:31:13.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.067+0000 7fb7597d7640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:13.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.067+0000 7fb7597d7640 10 monclient(hunting): authenticate will time out at 6121.248304s 2024-03-25T19:31:13.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.072+0000 7fb752ffd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fb7540a98f0 0x7fb7541d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:58792/0 (socket says 172.21.15.52:58792) 2024-03-25T19:31:13.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.072+0000 7fb752ffd640 1 -- 172.21.15.52:0/2253598856 learned_addr learned my addr 172.21.15.52:0/2253598856 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:13.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.075+0000 7fb73b7fe640 10 client.?.objecter ms_handle_connect 0x7fb7540a98f0 2024-03-25T19:31:13.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.075+0000 7fb73b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:13.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.087+0000 7fb73b7fe640 10 client.?.objecter ms_handle_connect 0x7fb75413f710 2024-03-25T19:31:13.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.087+0000 7fb73b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:13.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.089+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365268924 0 0) 0x7fb7540b1700 con 0x7fb7540a98f0 2024-03-25T19:31:13.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.089+0000 7fb73b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:13.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.089+0000 7fb73b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:13.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.089+0000 7fb73b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:13.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.089+0000 7fb73b7fe640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:31:13.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.089+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb740004eb0 con 0x7fb7540a98f0 2024-03-25T19:31:13.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.098+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942100595 0 0) 0x7fb7540b13d0 con 0x7fb75413f710 2024-03-25T19:31:13.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.098+0000 7fb73b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:13.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.098+0000 7fb73b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:13.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.098+0000 7fb73b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:13.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.098+0000 7fb73b7fe640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:31:13.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.099+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7540b1700 con 0x7fb75413f710 2024-03-25T19:31:13.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.102+0000 7fb73b7fe640 10 client.?.objecter ms_handle_connect 0x7fb7541430c0 2024-03-25T19:31:13.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.102+0000 7fb73b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:13.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.118+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 973723330 0 0) 0x7fb7540b1700 con 0x7fb75413f710 2024-03-25T19:31:13.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.118+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7540b13d0 con 0x7fb75413f710 2024-03-25T19:31:13.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.118+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb748001b20 con 0x7fb75413f710 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.118+0000 7fb73b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.118+0000 7fb73b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.118+0000 7fb73b7fe640 10 monclient(hunting): dump: 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:13.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.118+0000 7fb73b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:13.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.119+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4030946704 0 0) 0x7fb740004eb0 con 0x7fb7540a98f0 2024-03-25T19:31:13.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.119+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7540b1700 con 0x7fb7540a98f0 2024-03-25T19:31:13.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.119+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb744001a10 con 0x7fb7540a98f0 2024-03-25T19:31:13.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.119+0000 7fb73b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:13.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.119+0000 7fb73b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:13.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.119+0000 7fb73b7fe640 10 monclient(hunting): dump: 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:13.121 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:13.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.129+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322840711 0 0) 0x7fb7541d7a90 con 0x7fb7541430c0 2024-03-25T19:31:13.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.129+0000 7fb73b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:13.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.129+0000 7fb73b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:13.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.129+0000 7fb73b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:13.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.129+0000 7fb73b7fe640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:31:13.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.129+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb740004eb0 con 0x7fb7541430c0 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1324066765 0 0) 0x7fb7540b1700 con 0x7fb7540a98f0 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 >> v1:172.21.15.52:6790/0 conn(0x7fb75413f710 legacy=0x7fb7541d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 >> v1:172.21.15.52:6789/0 conn(0x7fb7541430c0 legacy=0x7fb75414d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb73b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb73b7fe640 1 monclient: found mon.b 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb73b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb73b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75414eb50 con 0x7fb7540a98f0 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb73b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb73b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb73b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:13.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb7597d7640 5 monclient: authenticate success, global_id 4399 2024-03-25T19:31:13.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb7597d7640 10 monclient: _renew_subs 2024-03-25T19:31:13.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb7597d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:13.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb7597d7640 1 -- 172.21.15.52:0/2253598856 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75414dc50 con 0x7fb7540a98f0 2024-03-25T19:31:13.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb7597d7640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:13.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb7597d7640 10 monclient: _renew_subs 2024-03-25T19:31:13.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb7597d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:13.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb7597d7640 1 -- 172.21.15.52:0/2253598856 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75414e290 con 0x7fb7540a98f0 2024-03-25T19:31:13.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.143+0000 7fb7597d7640 1 librados: init done 2024-03-25T19:31:13.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.145+0000 7fb7597d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:13.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.145+0000 7fb7597d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:13.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.145+0000 7fb7597d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:13.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.145+0000 7fb7597d7640 1 -- 172.21.15.52:0/2253598856 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7540b1560 con 0x7fb7540a98f0 2024-03-25T19:31:13.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.442+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb744001c70 con 0x7fb7540a98f0 2024-03-25T19:31:13.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.442+0000 7fb73b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:13.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.442+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb7440038c0 con 0x7fb7540a98f0 2024-03-25T19:31:13.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.442+0000 7fb73b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:13.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.442+0000 7fb73b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:13.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.442+0000 7fb73b7fe640 10 monclient: dump: 2024-03-25T19:31:13.443 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:13.443 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:13.444 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:13.444 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:13.444 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:13.444 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:13.444 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:13.444 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:13.444 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:13.444 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:13.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.442+0000 7fb7517fa640 4 set_mon_vals no callback set 2024-03-25T19:31:13.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.443+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fb744003ae0 con 0x7fb7540a98f0 2024-03-25T19:31:13.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.444+0000 7fb7527fc640 10 monclient: get_auth_request con 0x7fb74006df30 auth_method 0 2024-03-25T19:31:13.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.444+0000 7fb73b7fe640 10 client.4399.objecter ms_handle_connect 0x7fb74006df30 2024-03-25T19:31:13.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.449+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fb7440546e0 con 0x7fb7540a98f0 2024-03-25T19:31:13.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.449+0000 7fb73b7fe640 10 client.4399.objecter ms_dispatch 0x7fb754000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:13.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.449+0000 7fb73b7fe640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:13.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.449+0000 7fb73b7fe640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:13.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.449+0000 7fb73b7fe640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-25T19:31:13.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.449+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7440501f0 con 0x7fb7540a98f0 2024-03-25T19:31:13.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.449+0000 7fb73b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:13.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.449+0000 7fb73b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:13.693 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.693+0000 7fb7597d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:13.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.693+0000 7fb7597d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:13.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.693+0000 7fb7597d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:13.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.693+0000 7fb7597d7640 1 -- 172.21.15.52:0/2253598856 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7540b0d00 con 0x7fb7540a98f0 2024-03-25T19:31:13.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.731+0000 7fb73b7fe640 1 -- 172.21.15.52:0/2253598856 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fb744052120 con 0x7fb7540a98f0 2024-03-25T19:31:13.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.731+0000 7fb73b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:13.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.731+0000 7fb73b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:13.732 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:31:13.732 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:13.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 10 librados: watch_flush enter 2024-03-25T19:31:13.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 10 librados: watch_flush exit 2024-03-25T19:31:13.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-25T19:31:13.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 10 client.4399.objecter successfully canceled tick 2024-03-25T19:31:13.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 1 -- 172.21.15.52:0/2253598856 >> v1:172.21.15.52:6800/3967427147 conn(0x7fb74006df30 legacy=0x7fb7400703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 10 monclient: shutdown 2024-03-25T19:31:13.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:13.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 1 -- 172.21.15.52:0/2253598856 >> v1:172.21.15.179:6789/0 conn(0x7fb7540a98f0 legacy=0x7fb7541d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:13.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 1 -- 172.21.15.52:0/2253598856 shutdown_connections 2024-03-25T19:31:13.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 1 -- 172.21.15.52:0/2253598856 >> 172.21.15.52:0/2253598856 conn(0x7fb7540b7ea0 msgr2=0x7fb754074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:13.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 1 -- 172.21.15.52:0/2253598856 shutdown_connections 2024-03-25T19:31:13.736 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 1 -- 172.21.15.52:0/2253598856 wait complete. 2024-03-25T19:31:13.736 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:13.734+0000 7fb7597d7640 1 librados: shutdown 2024-03-25T19:31:13.746 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:20.553 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:20.668 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.669+0000 7f7345af6640 10 monclient: get_monmap_and_config 2024-03-25T19:31:20.668 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.669+0000 7f7345af6640 10 monclient: build_initial_monmap 2024-03-25T19:31:20.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.669+0000 7f7345af6640 10 monclient: monmap: 2024-03-25T19:31:20.669 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:20.669 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:20.669 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:31:20.669808+0000 2024-03-25T19:31:20.669 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:31:20.669808+0000 2024-03-25T19:31:20.669 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:20.669 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:20.669 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:20.669 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:31:20.669 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:31:20.669 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:20.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 1 Processor -- start 2024-03-25T19:31:20.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 1 -- start start 2024-03-25T19:31:20.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 10 monclient: init 2024-03-25T19:31:20.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:20.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:20.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 10 monclient(hunting): picked mon.noname-b con 0x7f73400a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:20.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 10 monclient(hunting): picked mon.noname-a con 0x7f734013f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:20.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 10 monclient(hunting): picked mon.noname-c con 0x7f73401430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:20.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7340142520 con 0x7f734013f710 2024-03-25T19:31:20.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73400b28a0 con 0x7f73401430c0 2024-03-25T19:31:20.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73400b2a30 con 0x7f73400a98f0 2024-03-25T19:31:20.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:20.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.670+0000 7f7345af6640 10 monclient(hunting): authenticate will time out at 6128.850863s 2024-03-25T19:31:20.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.693+0000 7f733f7fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f73400a98f0 0x7f73400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:46674/0 (socket says 172.21.15.52:46674) 2024-03-25T19:31:20.692 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.693+0000 7f733f7fe640 1 -- 172.21.15.52:0/2686229974 learned_addr learned my addr 172.21.15.52:0/2686229974 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:20.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.702+0000 7f733e7fc640 1 -- 172.21.15.52:0/2686229974 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837183637 0 0) 0x7f73400b2a30 con 0x7f73400a98f0 2024-03-25T19:31:20.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.702+0000 7f733e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.702+0000 7f733e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.702+0000 7f733e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.702+0000 7f733e7fc640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:31:20.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.702+0000 7f733e7fc640 1 -- 172.21.15.52:0/2686229974 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7328003520 con 0x7f73400a98f0 2024-03-25T19:31:20.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.704+0000 7f733e7fc640 1 -- 172.21.15.52:0/2686229974 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034752703 0 0) 0x7f7340142520 con 0x7f734013f710 2024-03-25T19:31:20.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.704+0000 7f733e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.704+0000 7f733e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.704+0000 7f733e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.704+0000 7f733e7fc640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:31:20.704 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.704+0000 7f733e7fc640 1 -- 172.21.15.52:0/2686229974 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73400b2a30 con 0x7f734013f710 2024-03-25T19:31:20.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.709+0000 7f733e7fc640 1 -- 172.21.15.52:0/2686229974 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1545847849 0 0) 0x7f73400b2a30 con 0x7f734013f710 2024-03-25T19:31:20.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.709+0000 7f733e7fc640 1 -- 172.21.15.52:0/2686229974 >> v1:172.21.15.52:6790/0 conn(0x7f73400a98f0 legacy=0x7f73400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.709+0000 7f733e7fc640 1 -- 172.21.15.52:0/2686229974 >> v1:172.21.15.179:6789/0 conn(0x7f73401430c0 legacy=0x7f734014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.709+0000 7f733e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.709+0000 7f733e7fc640 1 monclient: found mon.noname-a 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f733e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f733e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f733e7fc640 1 -- 172.21.15.52:0/2686229974 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73400b44c0 con 0x7f734013f710 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f733e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f733e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f733e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f733e7fc640 1 -- 172.21.15.52:0/2686229974 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7334001670 con 0x7f734013f710 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f733e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f733e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f733e7fc640 10 monclient: dump: 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:20.710 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:20.711 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:20.711 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:20.711 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:20.711 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:20.711 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:20.711 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:20.711 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:20.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f7345af6640 5 monclient: authenticate success, global_id 4392 2024-03-25T19:31:20.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.710+0000 7f7345af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.740+0000 7f733e7fc640 1 -- 172.21.15.52:0/2686229974 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73340017d0 con 0x7f734013f710 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.740+0000 7f733e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.740+0000 7f733e7fc640 1 -- 172.21.15.52:0/2686229974 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7334001ba0 con 0x7f734013f710 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.740+0000 7f733e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.740+0000 7f733e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.740+0000 7f733e7fc640 10 monclient: dump: 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:20.740 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:20.741 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:20.741 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:20.741 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:20.741 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:20.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.740+0000 7f7345af6640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:20.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.740+0000 7f7345af6640 4 set_mon_vals no callback set 2024-03-25T19:31:20.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.741+0000 7f7345af6640 10 monclient: shutdown 2024-03-25T19:31:20.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.741+0000 7f7345af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.741+0000 7f7345af6640 1 -- 172.21.15.52:0/2686229974 >> v1:172.21.15.52:6789/0 conn(0x7f734013f710 legacy=0x7f73401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.741+0000 7f7345af6640 1 -- 172.21.15.52:0/2686229974 shutdown_connections 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.741+0000 7f7345af6640 1 -- 172.21.15.52:0/2686229974 >> 172.21.15.52:0/2686229974 conn(0x7f73400b7ea0 msgr2=0x7f73400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.741+0000 7f7345af6640 1 -- 172.21.15.52:0/2686229974 shutdown_connections 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.741+0000 7f7345af6640 1 -- 172.21.15.52:0/2686229974 wait complete. 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.742+0000 7f7345af6640 10 monclient: build_initial_monmap 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.742+0000 7f7345af6640 10 monclient: monmap: 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:31:20.743 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:31:20.744 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:20.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.742+0000 7f7345af6640 1 librados: starting msgr at 2024-03-25T19:31:20.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.742+0000 7f7345af6640 1 librados: starting objecter 2024-03-25T19:31:20.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.742+0000 7f7345af6640 1 Processor -- start 2024-03-25T19:31:20.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 1 -- start start 2024-03-25T19:31:20.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 1 librados: setting wanted keys 2024-03-25T19:31:20.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 1 librados: calling monclient init 2024-03-25T19:31:20.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 10 monclient: init 2024-03-25T19:31:20.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:20.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:20.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 10 monclient(hunting): picked mon.noname-c con 0x7f73400a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:20.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 10 monclient(hunting): picked mon.noname-b con 0x7f734013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:20.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 10 monclient(hunting): picked mon.noname-a con 0x7f73401430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:20.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73400b7a60 con 0x7f73401430c0 2024-03-25T19:31:20.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73401dc070 con 0x7f734013f710 2024-03-25T19:31:20.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.743+0000 7f7345af6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734013b110 con 0x7f73400a98f0 2024-03-25T19:31:20.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.744+0000 7f7345af6640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:20.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.744+0000 7f7345af6640 10 monclient(hunting): authenticate will time out at 6128.923937s 2024-03-25T19:31:20.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.752+0000 7f733effd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f734013f710 0x7f73401db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:53358/0 (socket says 172.21.15.52:53358) 2024-03-25T19:31:20.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.752+0000 7f733effd640 1 -- 172.21.15.52:0/1417652518 learned_addr learned my addr 172.21.15.52:0/1417652518 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:20.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.757+0000 7f7327fff640 10 client.?.objecter ms_handle_connect 0x7f734013f710 2024-03-25T19:31:20.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.757+0000 7f7327fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:20.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.761+0000 7f7327fff640 10 client.?.objecter ms_handle_connect 0x7f73400a98f0 2024-03-25T19:31:20.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.761+0000 7f7327fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:20.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.764+0000 7f7327fff640 10 client.?.objecter ms_handle_connect 0x7f73401430c0 2024-03-25T19:31:20.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.764+0000 7f7327fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:20.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.767+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174459211 0 0) 0x7f73401dc070 con 0x7f734013f710 2024-03-25T19:31:20.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.767+0000 7f7327fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.767+0000 7f7327fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.767+0000 7f7327fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.767+0000 7f7327fff640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:31:20.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.767+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f732c004eb0 con 0x7f734013f710 2024-03-25T19:31:20.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.778+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1141042358 0 0) 0x7f732c004eb0 con 0x7f734013f710 2024-03-25T19:31:20.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.778+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73401dc070 con 0x7f734013f710 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.778+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7334001be0 con 0x7f734013f710 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.778+0000 7f7327fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.778+0000 7f7327fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.778+0000 7f7327fff640 10 monclient(hunting): dump: 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:20.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.778+0000 7f7327fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:20.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.779+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535190499 0 0) 0x7f734013b110 con 0x7f73400a98f0 2024-03-25T19:31:20.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.779+0000 7f7327fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.779+0000 7f7327fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.779+0000 7f7327fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.779+0000 7f7327fff640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:31:20.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.779+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f732c004eb0 con 0x7f73400a98f0 2024-03-25T19:31:20.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.784+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405822288 0 0) 0x7f73400b7a60 con 0x7f73401430c0 2024-03-25T19:31:20.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.784+0000 7f7327fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:20.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.784+0000 7f7327fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:20.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.784+0000 7f7327fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:20.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.784+0000 7f7327fff640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:31:20.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.784+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f734013b110 con 0x7f73401430c0 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.794+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1696635614 0 0) 0x7f73401dc070 con 0x7f734013f710 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 >> v1:172.21.15.52:6790/0 conn(0x7f73400a98f0 legacy=0x7f7340139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 >> v1:172.21.15.52:6789/0 conn(0x7f73401430c0 legacy=0x7f73401db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7327fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7327fff640 1 monclient: found mon.b 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7327fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7327fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734013b3f0 con 0x7f734013f710 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7327fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7327fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7327fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7345af6640 5 monclient: authenticate success, global_id 4405 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7345af6640 10 monclient: _renew_subs 2024-03-25T19:31:20.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7345af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:20.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7345af6640 1 -- 172.21.15.52:0/1417652518 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734013b6d0 con 0x7f734013f710 2024-03-25T19:31:20.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7345af6640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:20.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7345af6640 10 monclient: _renew_subs 2024-03-25T19:31:20.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7345af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:20.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7345af6640 1 -- 172.21.15.52:0/1417652518 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f734013a4c0 con 0x7f734013f710 2024-03-25T19:31:20.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.795+0000 7f7345af6640 1 librados: init done 2024-03-25T19:31:20.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.796+0000 7f7345af6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:20.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.796+0000 7f7345af6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:20.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.796+0000 7f7345af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:20.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.796+0000 7f7345af6640 1 -- 172.21.15.52:0/1417652518 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73400b3ef0 con 0x7f734013f710 2024-03-25T19:31:20.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.804+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73340038a0 con 0x7f734013f710 2024-03-25T19:31:20.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.804+0000 7f7327fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:20.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.804+0000 7f733dffb640 4 set_mon_vals no callback set 2024-03-25T19:31:20.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.806+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7334003c30 con 0x7f734013f710 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.806+0000 7f7327fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.806+0000 7f7327fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:20.806+0000 7f7327fff640 10 monclient: dump: 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:20.806 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:21.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.066+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f7334003e10 con 0x7f734013f710 2024-03-25T19:31:21.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.067+0000 7f733f7fe640 10 monclient: get_auth_request con 0x7f732c06de10 auth_method 0 2024-03-25T19:31:21.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.067+0000 7f7327fff640 10 client.4405.objecter ms_handle_connect 0x7f732c06de10 2024-03-25T19:31:21.069 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.069+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f7334054a20 con 0x7f734013f710 2024-03-25T19:31:21.069 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.069+0000 7f7327fff640 10 client.4405.objecter ms_dispatch 0x7f7340000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:21.069 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.069+0000 7f7327fff640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:21.069 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.069+0000 7f7327fff640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:21.069 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.069+0000 7f7327fff640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T19:31:21.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.073+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7334050360 con 0x7f734013f710 2024-03-25T19:31:21.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.073+0000 7f7327fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:21.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.073+0000 7f7327fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:21.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.334+0000 7f7345af6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:21.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.334+0000 7f7345af6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:21.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.334+0000 7f7345af6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:21.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.334+0000 7f7345af6640 1 -- 172.21.15.52:0/1417652518 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73400b7c10 con 0x7f734013f710 2024-03-25T19:31:21.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.380+0000 7f7327fff640 1 -- 172.21.15.52:0/1417652518 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f73400b7c10 con 0x7f734013f710 2024-03-25T19:31:21.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.380+0000 7f7327fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:21.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.380+0000 7f7327fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:21.380 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:31:21.380 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:21.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 10 librados: watch_flush enter 2024-03-25T19:31:21.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 10 librados: watch_flush exit 2024-03-25T19:31:21.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T19:31:21.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 10 client.4405.objecter successfully canceled tick 2024-03-25T19:31:21.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 1 -- 172.21.15.52:0/1417652518 >> v1:172.21.15.52:6800/3967427147 conn(0x7f732c06de10 legacy=0x7f732c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:21.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 10 monclient: shutdown 2024-03-25T19:31:21.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:21.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 1 -- 172.21.15.52:0/1417652518 >> v1:172.21.15.179:6789/0 conn(0x7f734013f710 legacy=0x7f73401db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:21.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 1 -- 172.21.15.52:0/1417652518 shutdown_connections 2024-03-25T19:31:21.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 1 -- 172.21.15.52:0/1417652518 >> 172.21.15.52:0/1417652518 conn(0x7f73400b7ea0 msgr2=0x7f7340074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:21.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 1 -- 172.21.15.52:0/1417652518 shutdown_connections 2024-03-25T19:31:21.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 1 -- 172.21.15.52:0/1417652518 wait complete. 2024-03-25T19:31:21.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:21.382+0000 7f7345af6640 1 librados: shutdown 2024-03-25T19:31:21.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:28.199 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:28.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.313+0000 7fb6376a8640 10 monclient: get_monmap_and_config 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.313+0000 7fb6376a8640 10 monclient: build_initial_monmap 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.313+0000 7fb6376a8640 10 monclient: monmap: 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:31:28.313964+0000 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:31:28.313964+0000 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:31:28.313 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.314+0000 7fb6376a8640 1 Processor -- start 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.314+0000 7fb6376a8640 1 -- start start 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.314+0000 7fb6376a8640 10 monclient: init 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.314+0000 7fb6376a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.314+0000 7fb6376a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.315+0000 7fb6376a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb6300a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.315+0000 7fb6376a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb63013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.315+0000 7fb6376a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb6301430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.315+0000 7fb6376a8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb630142520 con 0x7fb6301430c0 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.315+0000 7fb6376a8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6300b28a0 con 0x7fb6300a98f0 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.315+0000 7fb6376a8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6300b2a30 con 0x7fb63013f710 2024-03-25T19:31:28.315 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.315+0000 7fb6376a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:28.316 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.315+0000 7fb6376a8640 10 monclient(hunting): authenticate will time out at 6136.495115s 2024-03-25T19:31:28.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.319+0000 7fb635444640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fb6300a98f0 0x7fb6300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:59764/0 (socket says 172.21.15.52:59764) 2024-03-25T19:31:28.318 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.319+0000 7fb635444640 1 -- 172.21.15.52:0/434219876 learned_addr learned my addr 172.21.15.52:0/434219876 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:28.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.333+0000 7fb627fff640 1 -- 172.21.15.52:0/434219876 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810492329 0 0) 0x7fb6300b28a0 con 0x7fb6300a98f0 2024-03-25T19:31:28.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.333+0000 7fb627fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:28.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.333+0000 7fb627fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:28.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.333+0000 7fb627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:28.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.333+0000 7fb627fff640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:31:28.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.333+0000 7fb627fff640 1 -- 172.21.15.52:0/434219876 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb61c003520 con 0x7fb6300a98f0 2024-03-25T19:31:28.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.336+0000 7fb627fff640 1 -- 172.21.15.52:0/434219876 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011058563 0 0) 0x7fb630142520 con 0x7fb6301430c0 2024-03-25T19:31:28.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.336+0000 7fb627fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:28.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.336+0000 7fb627fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:28.335 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.336+0000 7fb627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:28.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.336+0000 7fb627fff640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:31:28.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.336+0000 7fb627fff640 1 -- 172.21.15.52:0/434219876 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6300b28a0 con 0x7fb6301430c0 2024-03-25T19:31:28.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 1 -- 172.21.15.52:0/434219876 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2816549155 0 0) 0x7fb61c003520 con 0x7fb6300a98f0 2024-03-25T19:31:28.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 1 -- 172.21.15.52:0/434219876 >> v1:172.21.15.52:6790/0 conn(0x7fb63013f710 legacy=0x7fb6301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 1 -- 172.21.15.52:0/434219876 >> v1:172.21.15.52:6789/0 conn(0x7fb6301430c0 legacy=0x7fb63014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:28.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 1 monclient: found mon.noname-c 2024-03-25T19:31:28.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:28.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:31:28.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 1 -- 172.21.15.52:0/434219876 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6300b44c0 con 0x7fb6300a98f0 2024-03-25T19:31:28.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:28.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:28.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 1 -- 172.21.15.52:0/434219876 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb62c001690 con 0x7fb6300a98f0 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb6376a8640 5 monclient: authenticate success, global_id 4408 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb6376a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb627fff640 10 monclient: dump: 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:28.344 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:28.345 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:28.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.343+0000 7fb6376a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.368+0000 7fb627fff640 1 -- 172.21.15.52:0/434219876 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb62c0017f0 con 0x7fb6300a98f0 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.368+0000 7fb627fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.368+0000 7fb627fff640 1 -- 172.21.15.52:0/434219876 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb62c001c00 con 0x7fb6300a98f0 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.368+0000 7fb6376a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.368+0000 7fb6376a8640 4 set_mon_vals no callback set 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.368+0000 7fb627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.368+0000 7fb627fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.368+0000 7fb627fff640 10 monclient: dump: 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.368 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.369 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:28.369 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:28.369 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:28.369 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:28.369 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:28.369 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:28.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.369+0000 7fb6376a8640 10 monclient: shutdown 2024-03-25T19:31:28.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.369+0000 7fb6376a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:28.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.369+0000 7fb6376a8640 1 -- 172.21.15.52:0/434219876 >> v1:172.21.15.179:6789/0 conn(0x7fb6300a98f0 legacy=0x7fb6300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.369+0000 7fb6376a8640 1 -- 172.21.15.52:0/434219876 shutdown_connections 2024-03-25T19:31:28.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.369+0000 7fb6376a8640 1 -- 172.21.15.52:0/434219876 >> 172.21.15.52:0/434219876 conn(0x7fb6300b7ea0 msgr2=0x7fb6300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:28.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.369+0000 7fb6376a8640 1 -- 172.21.15.52:0/434219876 shutdown_connections 2024-03-25T19:31:28.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.369+0000 7fb6376a8640 1 -- 172.21.15.52:0/434219876 wait complete. 2024-03-25T19:31:28.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.370+0000 7fb6376a8640 10 monclient: build_initial_monmap 2024-03-25T19:31:28.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.370+0000 7fb6376a8640 10 monclient: monmap: 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.370+0000 7fb6376a8640 1 librados: starting msgr at 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.370+0000 7fb6376a8640 1 librados: starting objecter 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.370+0000 7fb6376a8640 1 Processor -- start 2024-03-25T19:31:28.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.370+0000 7fb6376a8640 1 -- start start 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.370+0000 7fb6376a8640 1 librados: setting wanted keys 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.370+0000 7fb6376a8640 1 librados: calling monclient init 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.370+0000 7fb6376a8640 10 monclient: init 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.371+0000 7fb6376a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.371+0000 7fb6376a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.371+0000 7fb6376a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb6300a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.371+0000 7fb6376a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb63013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.371+0000 7fb6376a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb6301430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.371+0000 7fb6376a8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb630070670 con 0x7fb6300a98f0 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.371+0000 7fb6376a8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63006f4f0 con 0x7fb63013f710 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.371+0000 7fb6376a8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6301dfd50 con 0x7fb6301430c0 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.371+0000 7fb6376a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:28.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.371+0000 7fb6376a8640 10 monclient(hunting): authenticate will time out at 6136.551727s 2024-03-25T19:31:28.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.380+0000 7fb634c43640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fb63013f710 0x7fb6301df640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:59808/0 (socket says 172.21.15.52:59808) 2024-03-25T19:31:28.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.380+0000 7fb634c43640 1 -- 172.21.15.52:0/3286596526 learned_addr learned my addr 172.21.15.52:0/3286596526 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:28.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.386+0000 7fb625ffb640 10 client.?.objecter ms_handle_connect 0x7fb63013f710 2024-03-25T19:31:28.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.386+0000 7fb625ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:28.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.386+0000 7fb625ffb640 10 client.?.objecter ms_handle_connect 0x7fb6300a98f0 2024-03-25T19:31:28.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.386+0000 7fb625ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:28.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.393+0000 7fb625ffb640 10 client.?.objecter ms_handle_connect 0x7fb6301430c0 2024-03-25T19:31:28.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.393+0000 7fb625ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:28.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.393+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903017347 0 0) 0x7fb630070670 con 0x7fb6300a98f0 2024-03-25T19:31:28.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.393+0000 7fb625ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:28.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.393+0000 7fb625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:28.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.393+0000 7fb625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:28.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.393+0000 7fb625ffb640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:31:28.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.394+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb614004eb0 con 0x7fb6300a98f0 2024-03-25T19:31:28.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.399+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604457894 0 0) 0x7fb63006f4f0 con 0x7fb63013f710 2024-03-25T19:31:28.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.400+0000 7fb625ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:28.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.400+0000 7fb625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:28.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.400+0000 7fb625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:28.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.400+0000 7fb625ffb640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:31:28.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.400+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb630070670 con 0x7fb63013f710 2024-03-25T19:31:28.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.406+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288140284 0 0) 0x7fb6301dfd50 con 0x7fb6301430c0 2024-03-25T19:31:28.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.406+0000 7fb625ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:28.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.406+0000 7fb625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:28.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.406+0000 7fb625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:28.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.406+0000 7fb625ffb640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:31:28.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.406+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb63006f4f0 con 0x7fb6301430c0 2024-03-25T19:31:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.414+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3441052423 0 0) 0x7fb614004eb0 con 0x7fb6300a98f0 2024-03-25T19:31:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.414+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6301dfd50 con 0x7fb6300a98f0 2024-03-25T19:31:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.414+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb62c001b00 con 0x7fb6300a98f0 2024-03-25T19:31:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.414+0000 7fb625ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.414+0000 7fb625ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:28.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.414+0000 7fb625ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:28.414 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:28.414 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:28.414 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.415 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.415 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:28.415 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:28.415 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:28.415 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:28.415 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:28.415 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:28.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.414+0000 7fb625ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:28.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.419+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3393036384 0 0) 0x7fb630070670 con 0x7fb63013f710 2024-03-25T19:31:28.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.419+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb614004eb0 con 0x7fb63013f710 2024-03-25T19:31:28.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.419+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb6200019b0 con 0x7fb63013f710 2024-03-25T19:31:28.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.419+0000 7fb625ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:28.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.419+0000 7fb625ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:28.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.419+0000 7fb625ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:28.459 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:28.460 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:28.460 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.460 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.460 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:28.460 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:28.460 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:28.460 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:28.460 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:28.460 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.422+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1256879211 0 0) 0x7fb63006f4f0 con 0x7fb6301430c0 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.422+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb630070670 con 0x7fb6301430c0 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.422+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb628001b90 con 0x7fb6301430c0 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.422+0000 7fb625ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.422+0000 7fb625ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.422+0000 7fb625ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:28.462 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.437+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2750325056 0 0) 0x7fb6301dfd50 con 0x7fb6300a98f0 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.437+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 >> v1:172.21.15.52:6790/0 conn(0x7fb6301430c0 legacy=0x7fb630138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.437+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 >> v1:172.21.15.179:6789/0 conn(0x7fb63013f710 legacy=0x7fb6301df640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.437+0000 7fb625ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.437+0000 7fb625ffb640 1 monclient: found mon.a 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb625ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb625ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6301df120 con 0x7fb6300a98f0 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb625ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb625ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb625ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb6376a8640 5 monclient: authenticate success, global_id 4401 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb6376a8640 10 monclient: _renew_subs 2024-03-25T19:31:28.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb6376a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:28.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb6376a8640 1 -- 172.21.15.52:0/3286596526 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb63006e030 con 0x7fb6300a98f0 2024-03-25T19:31:28.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb6376a8640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:28.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb6376a8640 10 monclient: _renew_subs 2024-03-25T19:31:28.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb6376a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:28.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb6376a8640 1 -- 172.21.15.52:0/3286596526 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6301398e0 con 0x7fb6300a98f0 2024-03-25T19:31:28.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.438+0000 7fb6376a8640 1 librados: init done 2024-03-25T19:31:28.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.439+0000 7fb6376a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:28.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.439+0000 7fb6376a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:28.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.439+0000 7fb6376a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:28.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.439+0000 7fb6376a8640 1 -- 172.21.15.52:0/3286596526 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb630074cc0 con 0x7fb6300a98f0 2024-03-25T19:31:28.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.750+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb62c004a90 con 0x7fb6300a98f0 2024-03-25T19:31:28.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.750+0000 7fb625ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:28.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.750+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb62c004da0 con 0x7fb6300a98f0 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.750+0000 7fb625ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.750+0000 7fb625ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.750+0000 7fb625ffb640 10 monclient: dump: 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.750+0000 7fb6277fe640 4 set_mon_vals no callback set 2024-03-25T19:31:28.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.751+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fb62c01ad70 con 0x7fb6300a98f0 2024-03-25T19:31:28.752 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.752+0000 7fb634c43640 10 monclient: get_auth_request con 0x7fb61406e0b0 auth_method 0 2024-03-25T19:31:28.752 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.752+0000 7fb625ffb640 10 client.4401.objecter ms_handle_connect 0x7fb61406e0b0 2024-03-25T19:31:28.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.757+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fb62c054860 con 0x7fb6300a98f0 2024-03-25T19:31:28.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.757+0000 7fb625ffb640 10 client.4401.objecter ms_dispatch 0x7fb630000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:28.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.757+0000 7fb625ffb640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:28.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.757+0000 7fb625ffb640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:28.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.757+0000 7fb625ffb640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-25T19:31:28.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.757+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb62c050370 con 0x7fb6300a98f0 2024-03-25T19:31:28.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.757+0000 7fb625ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:28.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:28.757+0000 7fb625ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:29.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.023+0000 7fb6376a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:29.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.023+0000 7fb6376a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:29.022 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.023+0000 7fb6376a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:29.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.023+0000 7fb6376a8640 1 -- 172.21.15.52:0/3286596526 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb63006e330 con 0x7fb6300a98f0 2024-03-25T19:31:29.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.048+0000 7fb625ffb640 1 -- 172.21.15.52:0/3286596526 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fb62c0522a0 con 0x7fb6300a98f0 2024-03-25T19:31:29.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.048+0000 7fb625ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:29.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.048+0000 7fb625ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:29.048 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:31:29.048 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:29.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.050+0000 7fb6376a8640 10 librados: watch_flush enter 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.050+0000 7fb6376a8640 10 librados: watch_flush exit 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.050+0000 7fb6376a8640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.050+0000 7fb6376a8640 10 client.4401.objecter successfully canceled tick 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.051+0000 7fb6376a8640 1 -- 172.21.15.52:0/3286596526 >> v1:172.21.15.52:6800/3967427147 conn(0x7fb61406e0b0 legacy=0x7fb614070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.051+0000 7fb6376a8640 10 monclient: shutdown 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.051+0000 7fb6376a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.051+0000 7fb6376a8640 1 -- 172.21.15.52:0/3286596526 >> v1:172.21.15.52:6789/0 conn(0x7fb6300a98f0 legacy=0x7fb63006ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.051+0000 7fb6376a8640 1 -- 172.21.15.52:0/3286596526 shutdown_connections 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.051+0000 7fb6376a8640 1 -- 172.21.15.52:0/3286596526 >> 172.21.15.52:0/3286596526 conn(0x7fb6300b7ea0 msgr2=0x7fb630074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.051+0000 7fb6376a8640 1 -- 172.21.15.52:0/3286596526 shutdown_connections 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.051+0000 7fb6376a8640 1 -- 172.21.15.52:0/3286596526 wait complete. 2024-03-25T19:31:29.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:29.051+0000 7fb6376a8640 1 librados: shutdown 2024-03-25T19:31:29.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:35.868 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.985+0000 7f9997fc8640 10 monclient: get_monmap_and_config 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.985+0000 7f9997fc8640 10 monclient: build_initial_monmap 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.985+0000 7f9997fc8640 10 monclient: monmap: 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:31:35.986202+0000 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:31:35.986202+0000 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:31:35.985 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:31:35.986 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:35.986 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.986+0000 7f9997fc8640 1 Processor -- start 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.986+0000 7f9997fc8640 1 -- start start 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.986+0000 7f9997fc8640 10 monclient: init 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.987+0000 7f9997fc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.987+0000 7f9997fc8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.987+0000 7f9997fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f99900b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.987+0000 7f9997fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f99900b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.987+0000 7f9997fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f999014be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.987+0000 7f9997fc8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99900af940 con 0x7f999014be90 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.987+0000 7f9997fc8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99900b0a70 con 0x7f99900b4d60 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.987+0000 7f9997fc8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99900b0d50 con 0x7f99900b1730 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.987+0000 7f9997fc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:35.987 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.987+0000 7f9997fc8640 10 monclient(hunting): authenticate will time out at 6144.167483s 2024-03-25T19:31:35.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.990+0000 7f9995563640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f99900b1730 0x7f99900b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:39446/0 (socket says 172.21.15.52:39446) 2024-03-25T19:31:35.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:35.990+0000 7f9995563640 1 -- 172.21.15.52:0/1316944837 learned_addr learned my addr 172.21.15.52:0/1316944837 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:36.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.004+0000 7f9994d62640 1 -- 172.21.15.52:0/1316944837 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589683897 0 0) 0x7f99900b0a70 con 0x7f99900b4d60 2024-03-25T19:31:36.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.004+0000 7f9994d62640 10 monclient(hunting): _negotiate 2024-03-25T19:31:36.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.004+0000 7f9994d62640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:36.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.004+0000 7f9994d62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:36.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.004+0000 7f9994d62640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:31:36.004 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.004+0000 7f9994d62640 1 -- 172.21.15.52:0/1316944837 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997c003520 con 0x7f99900b4d60 2024-03-25T19:31:36.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.005+0000 7f9994d62640 1 -- 172.21.15.52:0/1316944837 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516689972 0 0) 0x7f99900b0d50 con 0x7f99900b1730 2024-03-25T19:31:36.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.005+0000 7f9994d62640 10 monclient(hunting): _negotiate 2024-03-25T19:31:36.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.005+0000 7f9994d62640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:36.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.005+0000 7f9994d62640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:36.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.005+0000 7f9994d62640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:31:36.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.005+0000 7f9994d62640 1 -- 172.21.15.52:0/1316944837 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99900b0a70 con 0x7f99900b1730 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 1 -- 172.21.15.52:0/1316944837 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1404293626 0 0) 0x7f99900b0a70 con 0x7f99900b1730 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 1 -- 172.21.15.52:0/1316944837 >> v1:172.21.15.179:6789/0 conn(0x7f99900b4d60 legacy=0x7f999013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 1 -- 172.21.15.52:0/1316944837 >> v1:172.21.15.52:6789/0 conn(0x7f999014be90 legacy=0x7f999014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 10 monclient: _finish_hunting 0 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 1 monclient: found mon.noname-b 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 1 -- 172.21.15.52:0/1316944837 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99900b1030 con 0x7f99900b1730 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 10 monclient: _finish_auth 0 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 10 monclient: _check_auth_tickets 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9997fc8640 5 monclient: authenticate success, global_id 4403 2024-03-25T19:31:36.017 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9997fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 1 -- 172.21.15.52:0/1316944837 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9980001670 con 0x7f99900b1730 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.016+0000 7f9994d62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.017+0000 7f9994d62640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.017+0000 7f9994d62640 10 monclient: dump: 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:36.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.017+0000 7f9997fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:36.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.042+0000 7f9994d62640 1 -- 172.21.15.52:0/1316944837 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99800017d0 con 0x7f99900b1730 2024-03-25T19:31:36.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.042+0000 7f9994d62640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:36.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.042+0000 7f9994d62640 1 -- 172.21.15.52:0/1316944837 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9980001b60 con 0x7f99900b1730 2024-03-25T19:31:36.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.042+0000 7f9997fc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:36.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.042+0000 7f9997fc8640 4 set_mon_vals no callback set 2024-03-25T19:31:36.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.042+0000 7f9994d62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:36.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.042+0000 7f9994d62640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:36.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.042+0000 7f9994d62640 10 monclient: dump: 2024-03-25T19:31:36.043 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:36.043 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:36.043 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:36.043 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:36.043 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:36.043 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:36.043 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:36.043 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:36.043 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:36.043 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:36.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.043+0000 7f9997fc8640 10 monclient: shutdown 2024-03-25T19:31:36.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.043+0000 7f9997fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:36.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.043+0000 7f9997fc8640 1 -- 172.21.15.52:0/1316944837 >> v1:172.21.15.52:6790/0 conn(0x7f99900b1730 legacy=0x7f99900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.043+0000 7f9997fc8640 1 -- 172.21.15.52:0/1316944837 shutdown_connections 2024-03-25T19:31:36.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.043+0000 7f9997fc8640 1 -- 172.21.15.52:0/1316944837 >> 172.21.15.52:0/1316944837 conn(0x7f99900617a0 msgr2=0x7f9990061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:36.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.043+0000 7f9997fc8640 1 -- 172.21.15.52:0/1316944837 shutdown_connections 2024-03-25T19:31:36.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.043+0000 7f9997fc8640 1 -- 172.21.15.52:0/1316944837 wait complete. 2024-03-25T19:31:36.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.044+0000 7f9997fc8640 10 monclient: build_initial_monmap 2024-03-25T19:31:36.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.044+0000 7f9997fc8640 10 monclient: monmap: 2024-03-25T19:31:36.045 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:36.045 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.044+0000 7f9997fc8640 1 librados: starting msgr at 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.044+0000 7f9997fc8640 1 librados: starting objecter 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.044+0000 7f9997fc8640 1 Processor -- start 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 1 -- start start 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 1 librados: setting wanted keys 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 1 librados: calling monclient init 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 10 monclient: init 2024-03-25T19:31:36.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:36.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:36.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 10 monclient(hunting): picked mon.noname-c con 0x7f99900b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:36.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 10 monclient(hunting): picked mon.noname-a con 0x7f99900b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:36.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 10 monclient(hunting): picked mon.noname-b con 0x7f999014be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:36.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99901383b0 con 0x7f99900b4d60 2024-03-25T19:31:36.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9990136160 con 0x7f999014be90 2024-03-25T19:31:36.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9990136d90 con 0x7f99900b1730 2024-03-25T19:31:36.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:36.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.045+0000 7f9997fc8640 10 monclient(hunting): authenticate will time out at 6144.225652s 2024-03-25T19:31:36.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.060+0000 7f9996565640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f999014be90 0x7f99901db620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:53794/0 (socket says 172.21.15.52:53794) 2024-03-25T19:31:36.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.060+0000 7f9996565640 1 -- 172.21.15.52:0/2570733157 learned_addr learned my addr 172.21.15.52:0/2570733157 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:36.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.062+0000 7f99867fc640 10 client.?.objecter ms_handle_connect 0x7f999014be90 2024-03-25T19:31:36.062 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.062+0000 7f99867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:36.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.065+0000 7f99867fc640 10 client.?.objecter ms_handle_connect 0x7f99900b4d60 2024-03-25T19:31:36.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.065+0000 7f99867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:36.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.074+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750975789 0 0) 0x7f9990136160 con 0x7f999014be90 2024-03-25T19:31:36.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.074+0000 7f99867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:36.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.074+0000 7f99867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:36.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.074+0000 7f99867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:36.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.074+0000 7f99867fc640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:31:36.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.074+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9974004eb0 con 0x7f999014be90 2024-03-25T19:31:36.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.077+0000 7f99867fc640 10 client.?.objecter ms_handle_connect 0x7f99900b1730 2024-03-25T19:31:36.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.077+0000 7f99867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:36.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.081+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919555983 0 0) 0x7f99901383b0 con 0x7f99900b4d60 2024-03-25T19:31:36.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.081+0000 7f99867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:36.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.081+0000 7f99867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:36.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.081+0000 7f99867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:36.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.081+0000 7f99867fc640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:31:36.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.081+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9990136160 con 0x7f99900b4d60 2024-03-25T19:31:36.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.085+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1859402785 0 0) 0x7f9974004eb0 con 0x7f999014be90 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.085+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99901383b0 con 0x7f999014be90 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.085+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9988001c90 con 0x7f999014be90 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.085+0000 7f99867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.085+0000 7f99867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.085+0000 7f99867fc640 10 monclient(hunting): dump: 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:36.085 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:36.086 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:36.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.085+0000 7f99867fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4041955122 0 0) 0x7f99901383b0 con 0x7f999014be90 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 >> v1:172.21.15.52:6790/0 conn(0x7f99900b1730 legacy=0x7f9990136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 >> v1:172.21.15.52:6789/0 conn(0x7f99900b4d60 legacy=0x7f99901daf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f99867fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f99867fc640 1 monclient: found mon.b 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f99867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f99867fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990136f20 con 0x7f999014be90 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f99867fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f99867fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f99867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:36.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f9997fc8640 5 monclient: authenticate success, global_id 4417 2024-03-25T19:31:36.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f9997fc8640 10 monclient: _renew_subs 2024-03-25T19:31:36.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f9997fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:36.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f9997fc8640 1 -- 172.21.15.52:0/2570733157 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9990143900 con 0x7f999014be90 2024-03-25T19:31:36.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f9997fc8640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:36.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f9997fc8640 10 monclient: _renew_subs 2024-03-25T19:31:36.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f9997fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:36.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f9997fc8640 1 -- 172.21.15.52:0/2570733157 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99901e3460 con 0x7f999014be90 2024-03-25T19:31:36.094 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.093+0000 7f9997fc8640 1 librados: init done 2024-03-25T19:31:36.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.094+0000 7f9997fc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:36.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.094+0000 7f9997fc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:36.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.094+0000 7f9997fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:36.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.094+0000 7f9997fc8640 1 -- 172.21.15.52:0/2570733157 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99900627b0 con 0x7f999014be90 2024-03-25T19:31:36.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.367+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9988003990 con 0x7f999014be90 2024-03-25T19:31:36.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.367+0000 7f99867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:36.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.367+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f9988003b40 con 0x7f999014be90 2024-03-25T19:31:36.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.367+0000 7f99867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:36.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.367+0000 7f99867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:36.367 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.367+0000 7f99867fc640 10 monclient: dump: 2024-03-25T19:31:36.367 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:36.367 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:36.367 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:36.368 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:36.368 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:36.368 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:36.368 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:36.368 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:36.368 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:36.368 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:36.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.367+0000 7f9987fff640 4 set_mon_vals no callback set 2024-03-25T19:31:36.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.369+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f9988003d60 con 0x7f999014be90 2024-03-25T19:31:36.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.369+0000 7f9995d64640 10 monclient: get_auth_request con 0x7f997406dd40 auth_method 0 2024-03-25T19:31:36.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.370+0000 7f99867fc640 10 client.4417.objecter ms_handle_connect 0x7f997406dd40 2024-03-25T19:31:36.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.373+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f9988054840 con 0x7f999014be90 2024-03-25T19:31:36.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.374+0000 7f99867fc640 10 client.4417.objecter ms_dispatch 0x7f9990000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:36.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.374+0000 7f99867fc640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:36.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.374+0000 7f99867fc640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:36.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.374+0000 7f99867fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-25T19:31:36.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.376+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9988050350 con 0x7f999014be90 2024-03-25T19:31:36.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.376+0000 7f99867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:36.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.376+0000 7f99867fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:36.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.625+0000 7f9997fc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:36.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.625+0000 7f9997fc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:36.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.625+0000 7f9997fc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:36.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.625+0000 7f9997fc8640 1 -- 172.21.15.52:0/2570733157 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99901362f0 con 0x7f999014be90 2024-03-25T19:31:36.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.797+0000 7f99867fc640 1 -- 172.21.15.52:0/2570733157 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f9988052280 con 0x7f999014be90 2024-03-25T19:31:36.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.797+0000 7f99867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:36.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.797+0000 7f99867fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:36.798 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:31:36.798 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:36.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 10 librados: watch_flush enter 2024-03-25T19:31:36.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 10 librados: watch_flush exit 2024-03-25T19:31:36.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-25T19:31:36.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 10 client.4417.objecter successfully canceled tick 2024-03-25T19:31:36.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 1 -- 172.21.15.52:0/2570733157 >> v1:172.21.15.52:6800/3967427147 conn(0x7f997406dd40 legacy=0x7f9974070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 10 monclient: shutdown 2024-03-25T19:31:36.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:36.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 1 -- 172.21.15.52:0/2570733157 >> v1:172.21.15.179:6789/0 conn(0x7f999014be90 legacy=0x7f99901db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:36.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 1 -- 172.21.15.52:0/2570733157 shutdown_connections 2024-03-25T19:31:36.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 1 -- 172.21.15.52:0/2570733157 >> 172.21.15.52:0/2570733157 conn(0x7f99900617a0 msgr2=0x7f9990061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:36.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 1 -- 172.21.15.52:0/2570733157 shutdown_connections 2024-03-25T19:31:36.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 1 -- 172.21.15.52:0/2570733157 wait complete. 2024-03-25T19:31:36.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:36.800+0000 7f9997fc8640 1 librados: shutdown 2024-03-25T19:31:36.812 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:43.619 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:43.736 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.736+0000 7fe431627640 10 monclient: get_monmap_and_config 2024-03-25T19:31:43.736 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.736+0000 7fe431627640 10 monclient: build_initial_monmap 2024-03-25T19:31:43.736 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.736+0000 7fe431627640 10 monclient: monmap: 2024-03-25T19:31:43.736 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:43.736 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:43.737 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:31:43.737434+0000 2024-03-25T19:31:43.737 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:31:43.737434+0000 2024-03-25T19:31:43.737 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:43.737 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:43.737 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:43.737 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:31:43.737 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:31:43.737 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:43.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.737+0000 7fe431627640 1 Processor -- start 2024-03-25T19:31:43.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.737+0000 7fe431627640 1 -- start start 2024-03-25T19:31:43.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.737+0000 7fe431627640 10 monclient: init 2024-03-25T19:31:43.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.738+0000 7fe431627640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:43.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.738+0000 7fe431627640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:43.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.738+0000 7fe431627640 10 monclient(hunting): picked mon.noname-c con 0x7fe42c0a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:43.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.738+0000 7fe431627640 10 monclient(hunting): picked mon.noname-a con 0x7fe42c13f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:43.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.738+0000 7fe431627640 10 monclient(hunting): picked mon.noname-b con 0x7fe42c1430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:43.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.738+0000 7fe431627640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c142520 con 0x7fe42c13f710 2024-03-25T19:31:43.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.738+0000 7fe431627640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c0b28a0 con 0x7fe42c0a98f0 2024-03-25T19:31:43.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.738+0000 7fe431627640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c0b2a30 con 0x7fe42c1430c0 2024-03-25T19:31:43.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.738+0000 7fe431627640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:43.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.738+0000 7fe431627640 10 monclient(hunting): authenticate will time out at 6151.918345s 2024-03-25T19:31:43.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.748+0000 7fe42a7fc640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fe42c13f710 0x7fe42c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:49768/0 (socket says 172.21.15.52:49768) 2024-03-25T19:31:43.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.748+0000 7fe42a7fc640 1 -- 172.21.15.52:0/4175644918 learned_addr learned my addr 172.21.15.52:0/4175644918 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:43.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.756+0000 7fe429ffb640 1 -- 172.21.15.52:0/4175644918 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554525705 0 0) 0x7fe42c142520 con 0x7fe42c13f710 2024-03-25T19:31:43.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.756+0000 7fe429ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.756+0000 7fe429ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.756+0000 7fe429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:43.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.756+0000 7fe429ffb640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:31:43.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.756+0000 7fe429ffb640 1 -- 172.21.15.52:0/4175644918 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe414003520 con 0x7fe42c13f710 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 1 -- 172.21.15.52:0/4175644918 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1377941410 0 0) 0x7fe414003520 con 0x7fe42c13f710 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 1 -- 172.21.15.52:0/4175644918 >> v1:172.21.15.52:6790/0 conn(0x7fe42c1430c0 legacy=0x7fe42c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 1 -- 172.21.15.52:0/4175644918 >> v1:172.21.15.179:6789/0 conn(0x7fe42c0a98f0 legacy=0x7fe42c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 1 monclient: found mon.noname-a 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 1 -- 172.21.15.52:0/4175644918 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c0b44c0 con 0x7fe42c13f710 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 1 -- 172.21.15.52:0/4175644918 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fe420001690 con 0x7fe42c13f710 2024-03-25T19:31:43.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe431627640 5 monclient: authenticate success, global_id 4410 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe431627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe429ffb640 10 monclient: dump: 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:43.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.760+0000 7fe431627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:43.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.773+0000 7fe429ffb640 1 -- 172.21.15.52:0/4175644918 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe420001860 con 0x7fe42c13f710 2024-03-25T19:31:43.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.773+0000 7fe429ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:43.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.773+0000 7fe431627640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:43.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.773+0000 7fe431627640 4 set_mon_vals no callback set 2024-03-25T19:31:43.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.774+0000 7fe431627640 10 monclient: shutdown 2024-03-25T19:31:43.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.774+0000 7fe431627640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:43.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.774+0000 7fe431627640 1 -- 172.21.15.52:0/4175644918 >> v1:172.21.15.52:6789/0 conn(0x7fe42c13f710 legacy=0x7fe42c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.774+0000 7fe431627640 1 -- 172.21.15.52:0/4175644918 shutdown_connections 2024-03-25T19:31:43.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.774+0000 7fe431627640 1 -- 172.21.15.52:0/4175644918 >> 172.21.15.52:0/4175644918 conn(0x7fe42c0b7ea0 msgr2=0x7fe42c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:43.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.774+0000 7fe431627640 1 -- 172.21.15.52:0/4175644918 shutdown_connections 2024-03-25T19:31:43.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.774+0000 7fe431627640 1 -- 172.21.15.52:0/4175644918 wait complete. 2024-03-25T19:31:43.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.775+0000 7fe431627640 10 monclient: build_initial_monmap 2024-03-25T19:31:43.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.775+0000 7fe431627640 10 monclient: monmap: 2024-03-25T19:31:43.775 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:43.775 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:43.775 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:31:43.775 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:31:43.775 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:43.775 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:31:43.776 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:43.776 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:31:43.776 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:31:43.776 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:43.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.775+0000 7fe431627640 1 librados: starting msgr at 2024-03-25T19:31:43.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.775+0000 7fe431627640 1 librados: starting objecter 2024-03-25T19:31:43.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.776+0000 7fe431627640 1 Processor -- start 2024-03-25T19:31:43.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.776+0000 7fe431627640 1 -- start start 2024-03-25T19:31:43.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.776+0000 7fe431627640 1 librados: setting wanted keys 2024-03-25T19:31:43.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.776+0000 7fe431627640 1 librados: calling monclient init 2024-03-25T19:31:43.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.776+0000 7fe431627640 10 monclient: init 2024-03-25T19:31:43.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.776+0000 7fe431627640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:43.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.776+0000 7fe431627640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:43.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.777+0000 7fe431627640 10 monclient(hunting): picked mon.noname-b con 0x7fe42c0a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:43.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.777+0000 7fe431627640 10 monclient(hunting): picked mon.noname-a con 0x7fe42c13f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:43.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.777+0000 7fe431627640 10 monclient(hunting): picked mon.noname-c con 0x7fe42c1430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:43.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.777+0000 7fe431627640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c0b0440 con 0x7fe42c13f710 2024-03-25T19:31:43.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.777+0000 7fe431627640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c0aef70 con 0x7fe42c0a98f0 2024-03-25T19:31:43.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.777+0000 7fe431627640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c0b0690 con 0x7fe42c1430c0 2024-03-25T19:31:43.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.777+0000 7fe431627640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:43.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.777+0000 7fe431627640 10 monclient(hunting): authenticate will time out at 6151.957386s 2024-03-25T19:31:43.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.782+0000 7fe42affd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fe42c0a98f0 0x7fe42c0b1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:58580/0 (socket says 172.21.15.52:58580) 2024-03-25T19:31:43.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.782+0000 7fe42affd640 1 -- 172.21.15.52:0/2943399365 learned_addr learned my addr 172.21.15.52:0/2943399365 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:43.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.786+0000 7fe4137fe640 10 client.?.objecter ms_handle_connect 0x7fe42c1430c0 2024-03-25T19:31:43.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.786+0000 7fe4137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:43.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.788+0000 7fe4137fe640 10 client.?.objecter ms_handle_connect 0x7fe42c0a98f0 2024-03-25T19:31:43.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.788+0000 7fe4137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:43.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.795+0000 7fe4137fe640 10 client.?.objecter ms_handle_connect 0x7fe42c13f710 2024-03-25T19:31:43.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.795+0000 7fe4137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:43.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.802+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494282055 0 0) 0x7fe42c0aef70 con 0x7fe42c0a98f0 2024-03-25T19:31:43.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.802+0000 7fe4137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.802+0000 7fe4137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.802+0000 7fe4137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:43.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.802+0000 7fe4137fe640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:31:43.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.802+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe418004eb0 con 0x7fe42c0a98f0 2024-03-25T19:31:43.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.807+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643077248 0 0) 0x7fe42c0b0690 con 0x7fe42c1430c0 2024-03-25T19:31:43.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.807+0000 7fe4137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.807+0000 7fe4137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.807+0000 7fe4137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:43.806 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.807+0000 7fe4137fe640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:31:43.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.807+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42c0aef70 con 0x7fe42c1430c0 2024-03-25T19:31:43.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.814+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152505032 0 0) 0x7fe42c0b0440 con 0x7fe42c13f710 2024-03-25T19:31:43.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.814+0000 7fe4137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.814+0000 7fe4137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.814+0000 7fe4137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:43.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.814+0000 7fe4137fe640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:31:43.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.814+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42c0b0690 con 0x7fe42c13f710 2024-03-25T19:31:43.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.818+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 449233180 0 0) 0x7fe418004eb0 con 0x7fe42c0a98f0 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.818+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe42c0b0440 con 0x7fe42c0a98f0 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.818+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fe41c001960 con 0x7fe42c0a98f0 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.818+0000 7fe4137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.818+0000 7fe4137fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.818+0000 7fe4137fe640 10 monclient(hunting): dump: 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:43.818 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:43.819 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:43.819 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:43.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.818+0000 7fe4137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:43.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.826+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1293173487 0 0) 0x7fe42c0aef70 con 0x7fe42c1430c0 2024-03-25T19:31:43.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.826+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe418004eb0 con 0x7fe42c1430c0 2024-03-25T19:31:43.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.826+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fe424001a70 con 0x7fe42c1430c0 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.826+0000 7fe4137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.826+0000 7fe4137fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.826+0000 7fe4137fe640 10 monclient(hunting): dump: 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:43.826 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1916886692 0 0) 0x7fe42c0b0440 con 0x7fe42c0a98f0 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 >> v1:172.21.15.52:6790/0 conn(0x7fe42c1430c0 legacy=0x7fe42c133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 >> v1:172.21.15.52:6789/0 conn(0x7fe42c13f710 legacy=0x7fe42c0ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe4137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe4137fe640 1 monclient: found mon.b 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe4137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe4137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c133780 con 0x7fe42c0a98f0 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe4137fe640 10 monclient: _finish_auth 0 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe4137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe4137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:43.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe431627640 5 monclient: authenticate success, global_id 4420 2024-03-25T19:31:43.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe431627640 10 monclient: _renew_subs 2024-03-25T19:31:43.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe431627640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:43.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe431627640 1 -- 172.21.15.52:0/2943399365 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42c1339d0 con 0x7fe42c0a98f0 2024-03-25T19:31:43.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe431627640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:43.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe431627640 10 monclient: _renew_subs 2024-03-25T19:31:43.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe431627640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:43.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe431627640 1 -- 172.21.15.52:0/2943399365 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42c0ae9a0 con 0x7fe42c0a98f0 2024-03-25T19:31:43.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.828+0000 7fe431627640 1 librados: init done 2024-03-25T19:31:43.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.829+0000 7fe431627640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:43.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.829+0000 7fe431627640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:43.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.829+0000 7fe431627640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:43.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.829+0000 7fe431627640 1 -- 172.21.15.52:0/2943399365 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe42c0627b0 con 0x7fe42c0a98f0 2024-03-25T19:31:43.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.835+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe41c0035f0 con 0x7fe42c0a98f0 2024-03-25T19:31:43.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.835+0000 7fe4137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:43.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.835+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fe41c003940 con 0x7fe42c0a98f0 2024-03-25T19:31:43.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.835+0000 7fe4137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.835+0000 7fe4137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:43.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.835+0000 7fe4137fe640 10 monclient: dump: 2024-03-25T19:31:43.836 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:43.836 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:43.836 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:43.836 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:43.836 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.836 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:43.836 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:43.836 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:43.836 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:43.836 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:43.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:43.835+0000 7fe4297fa640 4 set_mon_vals no callback set 2024-03-25T19:31:44.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.106+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fe41c003aa0 con 0x7fe42c0a98f0 2024-03-25T19:31:44.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.107+0000 7fe42a7fc640 10 monclient: get_auth_request con 0x7fe41806df70 auth_method 0 2024-03-25T19:31:44.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.107+0000 7fe4137fe640 10 client.4420.objecter ms_handle_connect 0x7fe41806df70 2024-03-25T19:31:44.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.110+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fe41c054760 con 0x7fe42c0a98f0 2024-03-25T19:31:44.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.110+0000 7fe4137fe640 10 client.4420.objecter ms_dispatch 0x7fe42c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:44.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.110+0000 7fe4137fe640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:44.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.110+0000 7fe4137fe640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:44.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.110+0000 7fe4137fe640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T19:31:44.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.113+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe41c050270 con 0x7fe42c0a98f0 2024-03-25T19:31:44.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.113+0000 7fe4137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:44.113 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.113+0000 7fe4137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:44.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.332+0000 7fe431627640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:44.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.332+0000 7fe431627640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:44.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.332+0000 7fe431627640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:31:44.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.332+0000 7fe431627640 1 -- 172.21.15.52:0/2943399365 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe42c13fdc0 con 0x7fe42c0a98f0 2024-03-25T19:31:44.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.352+0000 7fe4137fe640 1 -- 172.21.15.52:0/2943399365 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fe41c0521a0 con 0x7fe42c0a98f0 2024-03-25T19:31:44.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.352+0000 7fe4137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:44.351 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.352+0000 7fe4137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:44.352 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:31:44.352 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:44.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 10 librados: watch_flush enter 2024-03-25T19:31:44.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 10 librados: watch_flush exit 2024-03-25T19:31:44.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T19:31:44.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 10 client.4420.objecter successfully canceled tick 2024-03-25T19:31:44.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 1 -- 172.21.15.52:0/2943399365 >> v1:172.21.15.52:6800/3967427147 conn(0x7fe41806df70 legacy=0x7fe418070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:44.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 10 monclient: shutdown 2024-03-25T19:31:44.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:44.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 1 -- 172.21.15.52:0/2943399365 >> v1:172.21.15.179:6789/0 conn(0x7fe42c0a98f0 legacy=0x7fe42c0b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:44.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 1 -- 172.21.15.52:0/2943399365 shutdown_connections 2024-03-25T19:31:44.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 1 -- 172.21.15.52:0/2943399365 >> 172.21.15.52:0/2943399365 conn(0x7fe42c0b7ea0 msgr2=0x7fe42c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:44.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 1 -- 172.21.15.52:0/2943399365 shutdown_connections 2024-03-25T19:31:44.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.354+0000 7fe431627640 1 -- 172.21.15.52:0/2943399365 wait complete. 2024-03-25T19:31:44.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:44.355+0000 7fe431627640 1 librados: shutdown 2024-03-25T19:31:44.366 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:46.551 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:04:40.150 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:46.562 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:04:40.161 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:46.589 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:04:40.188 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:46.617 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:04:40.215 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:46.619 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:04:40.218 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:46.628 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:04:40.170 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:46.632 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:04:40.173 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:46.650 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:04:40.249 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:46.670 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:04:40.212 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:51.174 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:51.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.287+0000 7f48261c6640 10 monclient: get_monmap_and_config 2024-03-25T19:31:51.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.287+0000 7f48261c6640 10 monclient: build_initial_monmap 2024-03-25T19:31:51.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.287+0000 7f48261c6640 10 monclient: monmap: 2024-03-25T19:31:51.287 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:51.287 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:51.287 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:31:51.288487+0000 2024-03-25T19:31:51.287 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:31:51.288487+0000 2024-03-25T19:31:51.287 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:51.288 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:51.288 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:51.288 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:31:51.288 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:31:51.288 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:51.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.288+0000 7f48261c6640 1 Processor -- start 2024-03-25T19:31:51.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.288+0000 7f48261c6640 1 -- start start 2024-03-25T19:31:51.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.288+0000 7f48261c6640 10 monclient: init 2024-03-25T19:31:51.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.289+0000 7f48261c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:51.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.289+0000 7f48261c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:51.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.289+0000 7f48261c6640 10 monclient(hunting): picked mon.noname-b con 0x7f48200b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:51.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.289+0000 7f48261c6640 10 monclient(hunting): picked mon.noname-c con 0x7f48200b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:51.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.289+0000 7f48261c6640 10 monclient(hunting): picked mon.noname-a con 0x7f482014be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:51.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.289+0000 7f48261c6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48200af940 con 0x7f482014be90 2024-03-25T19:31:51.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.289+0000 7f48261c6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48200b0a70 con 0x7f48200b1730 2024-03-25T19:31:51.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.289+0000 7f48261c6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48200b0d50 con 0x7f48200b4d60 2024-03-25T19:31:51.290 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.289+0000 7f48261c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:51.290 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.289+0000 7f48261c6640 10 monclient(hunting): authenticate will time out at 6159.469565s 2024-03-25T19:31:51.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.300+0000 7f481ffff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f482014be90 0x7f482014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:50958/0 (socket says 172.21.15.52:50958) 2024-03-25T19:31:51.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.300+0000 7f481ffff640 1 -- 172.21.15.52:0/1596487621 learned_addr learned my addr 172.21.15.52:0/1596487621 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:51.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.314+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567668114 0 0) 0x7f48200b0d50 con 0x7f48200b4d60 2024-03-25T19:31:51.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.314+0000 7f481e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:51.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.314+0000 7f481e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:51.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.314+0000 7f481e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:51.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.314+0000 7f481e7fc640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:31:51.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.314+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4808003520 con 0x7f48200b4d60 2024-03-25T19:31:51.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.320+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744441675 0 0) 0x7f48200b0a70 con 0x7f48200b1730 2024-03-25T19:31:51.319 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.320+0000 7f481e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:51.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.320+0000 7f481e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:51.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.320+0000 7f481e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:51.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.320+0000 7f481e7fc640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:31:51.320 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.320+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48200b0d50 con 0x7f48200b1730 2024-03-25T19:31:51.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.330+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698414918 0 0) 0x7f48200af940 con 0x7f482014be90 2024-03-25T19:31:51.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.331+0000 7f481e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:51.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.331+0000 7f481e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:51.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.331+0000 7f481e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:51.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.331+0000 7f481e7fc640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:31:51.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.331+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48200b0a70 con 0x7f482014be90 2024-03-25T19:31:51.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.332+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1136140695 0 0) 0x7f4808003520 con 0x7f48200b4d60 2024-03-25T19:31:51.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.332+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 >> v1:172.21.15.179:6789/0 conn(0x7f48200b1730 legacy=0x7f48200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.332+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 >> v1:172.21.15.52:6789/0 conn(0x7f482014be90 legacy=0x7f482014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.332+0000 7f481e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.332+0000 7f481e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.332+0000 7f481e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.332+0000 7f481e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.332+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48200b1030 con 0x7f48200b4d60 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.332+0000 7f481e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.333+0000 7f481e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.333+0000 7f481e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.333+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4810001690 con 0x7f48200b4d60 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.333+0000 7f48261c6640 5 monclient: authenticate success, global_id 4412 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.333+0000 7f481e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.333+0000 7f481e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.333+0000 7f481e7fc640 10 monclient: dump: 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:51.333 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:51.334 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:51.334 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:51.334 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:51.334 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:51.334 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:51.334 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:51.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.333+0000 7f48261c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:51.338 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.338+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48100017f0 con 0x7f48200b4d60 2024-03-25T19:31:51.338 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.338+0000 7f481e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:51.338 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.338+0000 7f481e7fc640 1 -- 172.21.15.52:0/1596487621 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4810001c00 con 0x7f48200b4d60 2024-03-25T19:31:51.338 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.338+0000 7f481e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:51.338 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.338+0000 7f481e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:51.338 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.338+0000 7f481e7fc640 10 monclient: dump: 2024-03-25T19:31:51.338 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:51.339 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:51.339 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:51.339 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:51.339 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:51.339 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:51.339 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:51.339 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:51.339 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:51.339 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:51.339 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.338+0000 7f48261c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:51.339 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.338+0000 7f48261c6640 4 set_mon_vals no callback set 2024-03-25T19:31:51.341 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.339+0000 7f48261c6640 10 monclient: shutdown 2024-03-25T19:31:51.341 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.339+0000 7f48261c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:51.341 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.339+0000 7f48261c6640 1 -- 172.21.15.52:0/1596487621 >> v1:172.21.15.52:6790/0 conn(0x7f48200b4d60 legacy=0x7f482013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.339+0000 7f48261c6640 1 -- 172.21.15.52:0/1596487621 shutdown_connections 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.339+0000 7f48261c6640 1 -- 172.21.15.52:0/1596487621 >> 172.21.15.52:0/1596487621 conn(0x7f48200617a0 msgr2=0x7f4820061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.339+0000 7f48261c6640 1 -- 172.21.15.52:0/1596487621 shutdown_connections 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.339+0000 7f48261c6640 1 -- 172.21.15.52:0/1596487621 wait complete. 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.340+0000 7f48261c6640 10 monclient: build_initial_monmap 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.340+0000 7f48261c6640 10 monclient: monmap: 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:51.342 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.340+0000 7f48261c6640 1 librados: starting msgr at 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.340+0000 7f48261c6640 1 librados: starting objecter 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.341+0000 7f48261c6640 1 Processor -- start 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.341+0000 7f48261c6640 1 -- start start 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.341+0000 7f48261c6640 1 librados: setting wanted keys 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.341+0000 7f48261c6640 1 librados: calling monclient init 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.341+0000 7f48261c6640 10 monclient: init 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.341+0000 7f48261c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.341+0000 7f48261c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.341+0000 7f48261c6640 10 monclient(hunting): picked mon.noname-a con 0x7f48200b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.341+0000 7f48261c6640 10 monclient(hunting): picked mon.noname-b con 0x7f48200b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:51.343 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.342+0000 7f48261c6640 10 monclient(hunting): picked mon.noname-c con 0x7f482014be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:51.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.342+0000 7f48261c6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48201d7980 con 0x7f48200b1730 2024-03-25T19:31:51.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.342+0000 7f48261c6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48201420a0 con 0x7f48200b4d60 2024-03-25T19:31:51.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.342+0000 7f48261c6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48201432c0 con 0x7f482014be90 2024-03-25T19:31:51.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.342+0000 7f48261c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:51.344 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.342+0000 7f48261c6640 10 monclient(hunting): authenticate will time out at 6159.521983s 2024-03-25T19:31:51.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.353+0000 7f481f7fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f48200b1730 0x7f4820141990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:51014/0 (socket says 172.21.15.52:51014) 2024-03-25T19:31:51.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.354+0000 7f481f7fe640 1 -- 172.21.15.52:0/3677469467 learned_addr learned my addr 172.21.15.52:0/3677469467 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:51.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.356+0000 7f4807fff640 10 client.?.objecter ms_handle_connect 0x7f48200b1730 2024-03-25T19:31:51.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.356+0000 7f4807fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:51.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.358+0000 7f4807fff640 10 client.?.objecter ms_handle_connect 0x7f48200b4d60 2024-03-25T19:31:51.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.358+0000 7f4807fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:51.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.363+0000 7f4807fff640 10 client.?.objecter ms_handle_connect 0x7f482014be90 2024-03-25T19:31:51.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.363+0000 7f4807fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:51.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.365+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841171240 0 0) 0x7f48201d7980 con 0x7f48200b1730 2024-03-25T19:31:51.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.365+0000 7f4807fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:51.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.365+0000 7f4807fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:51.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.365+0000 7f4807fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:51.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.365+0000 7f4807fff640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:31:51.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.365+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480c004eb0 con 0x7f48200b1730 2024-03-25T19:31:51.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.374+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063672691 0 0) 0x7f48201420a0 con 0x7f48200b4d60 2024-03-25T19:31:51.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.374+0000 7f4807fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:51.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.374+0000 7f4807fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:51.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.374+0000 7f4807fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:51.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.374+0000 7f4807fff640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:31:51.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.374+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48201d7980 con 0x7f48200b4d60 2024-03-25T19:31:51.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.375+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555210618 0 0) 0x7f48201432c0 con 0x7f482014be90 2024-03-25T19:31:51.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.375+0000 7f4807fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:51.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.375+0000 7f4807fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:51.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.375+0000 7f4807fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:51.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.375+0000 7f4807fff640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:31:51.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.375+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48201420a0 con 0x7f482014be90 2024-03-25T19:31:51.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.376+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1061541866 0 0) 0x7f480c004eb0 con 0x7f48200b1730 2024-03-25T19:31:51.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.376+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48201432c0 con 0x7f48200b1730 2024-03-25T19:31:51.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.376+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4810001bc0 con 0x7f48200b1730 2024-03-25T19:31:51.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.376+0000 7f4807fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:51.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.376+0000 7f4807fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.376+0000 7f4807fff640 10 monclient(hunting): dump: 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:51.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.376+0000 7f4807fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:51.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1741456109 0 0) 0x7f48201432c0 con 0x7f48200b1730 2024-03-25T19:31:51.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 >> v1:172.21.15.52:6790/0 conn(0x7f482014be90 legacy=0x7f48201d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 >> v1:172.21.15.179:6789/0 conn(0x7f48200b4d60 legacy=0x7f4820138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f4807fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:51.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f4807fff640 1 monclient: found mon.a 2024-03-25T19:31:51.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f4807fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:51.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f4807fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:51.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4820143940 con 0x7f48200b1730 2024-03-25T19:31:51.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f4807fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:51.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f4807fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:51.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f4807fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:51.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f48261c6640 5 monclient: authenticate success, global_id 4419 2024-03-25T19:31:51.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f48261c6640 10 monclient: _renew_subs 2024-03-25T19:31:51.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f48261c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:51.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f48261c6640 1 -- 172.21.15.52:0/3677469467 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4820138a70 con 0x7f48200b1730 2024-03-25T19:31:51.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f48261c6640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:51.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f48261c6640 10 monclient: _renew_subs 2024-03-25T19:31:51.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f48261c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:51.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f48261c6640 1 -- 172.21.15.52:0/3677469467 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48201d8700 con 0x7f48200b1730 2024-03-25T19:31:51.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.384+0000 7f48261c6640 1 librados: init done 2024-03-25T19:31:51.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.385+0000 7f48261c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:51.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.385+0000 7f48261c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:51.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.385+0000 7f48261c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:51.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.385+0000 7f48261c6640 1 -- 172.21.15.52:0/3677469467 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4820143130 con 0x7f48200b1730 2024-03-25T19:31:51.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.709+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48100011f0 con 0x7f48200b1730 2024-03-25T19:31:51.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.709+0000 7f4807fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:51.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.709+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4810003b00 con 0x7f48200b1730 2024-03-25T19:31:51.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.709+0000 7f4807fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:51.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.709+0000 7f4807fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:51.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.709+0000 7f4807fff640 10 monclient: dump: 2024-03-25T19:31:51.709 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:51.709 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:51.709 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:51.709 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:51.709 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:51.710 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:51.710 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:51.710 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:51.710 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:51.710 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:51.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.709+0000 7f481dffb640 4 set_mon_vals no callback set 2024-03-25T19:31:51.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.710+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f4810003d20 con 0x7f48200b1730 2024-03-25T19:31:51.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.710+0000 7f481effd640 10 monclient: get_auth_request con 0x7f480c06ddf0 auth_method 0 2024-03-25T19:31:51.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.711+0000 7f4807fff640 10 client.4419.objecter ms_handle_connect 0x7f480c06ddf0 2024-03-25T19:31:51.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.716+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f48100548d0 con 0x7f48200b1730 2024-03-25T19:31:51.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.716+0000 7f4807fff640 10 client.4419.objecter ms_dispatch 0x7f4820000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:51.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.716+0000 7f4807fff640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:51.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.716+0000 7f4807fff640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:51.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.716+0000 7f4807fff640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-25T19:31:51.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.716+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48100503e0 con 0x7f48200b1730 2024-03-25T19:31:51.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.716+0000 7f4807fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:51.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.716+0000 7f4807fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:51.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.949+0000 7f48261c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:51.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.949+0000 7f48261c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:51.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.949+0000 7f48261c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:51.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.949+0000 7f48261c6640 1 -- 172.21.15.52:0/3677469467 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48201401f0 con 0x7f48200b1730 2024-03-25T19:31:51.966 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.966+0000 7f4807fff640 1 -- 172.21.15.52:0/3677469467 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f4810052310 con 0x7f48200b1730 2024-03-25T19:31:51.966 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.966+0000 7f4807fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:51.966 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.966+0000 7f4807fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:51.966 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:31:51.966 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:51.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.968+0000 7f48261c6640 10 librados: watch_flush enter 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.968+0000 7f48261c6640 10 librados: watch_flush exit 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.968+0000 7f48261c6640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.968+0000 7f48261c6640 10 client.4419.objecter successfully canceled tick 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.969+0000 7f48261c6640 1 -- 172.21.15.52:0/3677469467 >> v1:172.21.15.52:6800/3967427147 conn(0x7f480c06ddf0 legacy=0x7f480c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.969+0000 7f48261c6640 10 monclient: shutdown 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.969+0000 7f48261c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.969+0000 7f48261c6640 1 -- 172.21.15.52:0/3677469467 >> v1:172.21.15.52:6789/0 conn(0x7f48200b1730 legacy=0x7f4820141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.969+0000 7f48261c6640 1 -- 172.21.15.52:0/3677469467 shutdown_connections 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.969+0000 7f48261c6640 1 -- 172.21.15.52:0/3677469467 >> 172.21.15.52:0/3677469467 conn(0x7f48200617a0 msgr2=0x7f4820061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.969+0000 7f48261c6640 1 -- 172.21.15.52:0/3677469467 shutdown_connections 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.969+0000 7f48261c6640 1 -- 172.21.15.52:0/3677469467 wait complete. 2024-03-25T19:31:51.969 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:51.969+0000 7f48261c6640 1 librados: shutdown 2024-03-25T19:31:51.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:58.788 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:58.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.903+0000 7f8ac5d1c640 10 monclient: get_monmap_and_config 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.903+0000 7f8ac5d1c640 10 monclient: build_initial_monmap 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.903+0000 7f8ac5d1c640 10 monclient: monmap: 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:31:58.904847+0000 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:31:58.904847+0000 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:31:58.904 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:58.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.904+0000 7f8ac5d1c640 1 Processor -- start 2024-03-25T19:31:58.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.904+0000 7f8ac5d1c640 1 -- start start 2024-03-25T19:31:58.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.904+0000 7f8ac5d1c640 10 monclient: init 2024-03-25T19:31:58.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.904+0000 7f8ac5d1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:58.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.904+0000 7f8ac5d1c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:58.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.905+0000 7f8ac5d1c640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac00a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:58.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.905+0000 7f8ac5d1c640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:58.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.905+0000 7f8ac5d1c640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac01430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:58.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.905+0000 7f8ac5d1c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac0142520 con 0x7f8ac00a98f0 2024-03-25T19:31:58.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.905+0000 7f8ac5d1c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac00b28a0 con 0x7f8ac01430c0 2024-03-25T19:31:58.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.905+0000 7f8ac5d1c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac00b2a30 con 0x7f8ac013f710 2024-03-25T19:31:58.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.905+0000 7f8ac5d1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:58.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.905+0000 7f8ac5d1c640 10 monclient(hunting): authenticate will time out at 6167.086007s 2024-03-25T19:31:58.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.909+0000 7f8abeffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f8ac013f710 0x7f8ac01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37470/0 (socket says 172.21.15.52:37470) 2024-03-25T19:31:58.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.909+0000 7f8abeffd640 1 -- 172.21.15.52:0/3835695217 learned_addr learned my addr 172.21.15.52:0/3835695217 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:58.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.921+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73378543 0 0) 0x7f8ac00b2a30 con 0x7f8ac013f710 2024-03-25T19:31:58.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.921+0000 7f8abe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.921+0000 7f8abe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.921+0000 7f8abe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.921+0000 7f8abe7fc640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:31:58.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.921+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aa8003520 con 0x7f8ac013f710 2024-03-25T19:31:58.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.923+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289699285 0 0) 0x7f8ac0142520 con 0x7f8ac00a98f0 2024-03-25T19:31:58.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.923+0000 7f8abe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.923+0000 7f8abe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.923+0000 7f8abe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.923+0000 7f8abe7fc640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:31:58.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.923+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac00b2a30 con 0x7f8ac00a98f0 2024-03-25T19:31:58.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.931+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668347590 0 0) 0x7f8ac00b28a0 con 0x7f8ac01430c0 2024-03-25T19:31:58.931 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.931+0000 7f8abe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.931+0000 7f8abe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.931+0000 7f8abe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.931+0000 7f8abe7fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:31:58.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.931+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac0142520 con 0x7f8ac01430c0 2024-03-25T19:31:58.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3744326800 0 0) 0x7f8aa8003520 con 0x7f8ac013f710 2024-03-25T19:31:58.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 >> v1:172.21.15.179:6789/0 conn(0x7f8ac01430c0 legacy=0x7f8ac014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 >> v1:172.21.15.52:6789/0 conn(0x7f8ac00a98f0 legacy=0x7f8ac00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:58.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 1 monclient: found mon.noname-b 2024-03-25T19:31:58.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:58.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:31:58.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac00b44c0 con 0x7f8ac013f710 2024-03-25T19:31:58.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8ab4001690 con 0x7f8ac013f710 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8abe7fc640 10 monclient: dump: 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:58.937 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:58.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8ac5d1c640 5 monclient: authenticate success, global_id 4418 2024-03-25T19:31:58.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.935+0000 7f8ac5d1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:58.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ab40017f0 con 0x7f8ac013f710 2024-03-25T19:31:58.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8abe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:58.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8abe7fc640 1 -- 172.21.15.52:0/3835695217 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8ab4001c00 con 0x7f8ac013f710 2024-03-25T19:31:58.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8abe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8abe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:58.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8abe7fc640 10 monclient: dump: 2024-03-25T19:31:58.948 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:58.948 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:58.948 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:58.949 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:58.949 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.949 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:58.949 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:58.949 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:58.949 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:58.950 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:58.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8ac5d1c640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:58.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8ac5d1c640 4 set_mon_vals no callback set 2024-03-25T19:31:58.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8ac5d1c640 10 monclient: shutdown 2024-03-25T19:31:58.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8ac5d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:58.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/3835695217 >> v1:172.21.15.52:6790/0 conn(0x7f8ac013f710 legacy=0x7f8ac01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/3835695217 shutdown_connections 2024-03-25T19:31:58.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/3835695217 >> 172.21.15.52:0/3835695217 conn(0x7f8ac00b7ea0 msgr2=0x7f8ac00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:58.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.947+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/3835695217 shutdown_connections 2024-03-25T19:31:58.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.948+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/3835695217 wait complete. 2024-03-25T19:31:58.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.948+0000 7f8ac5d1c640 10 monclient: build_initial_monmap 2024-03-25T19:31:58.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.948+0000 7f8ac5d1c640 10 monclient: monmap: 2024-03-25T19:31:58.957 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:31:58.957 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:58.957 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:31:58.957 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:31:58.957 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:58.957 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:31:58.958 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:31:58.958 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:31:58.958 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:31:58.958 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:58.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.949+0000 7f8ac5d1c640 1 librados: starting msgr at 2024-03-25T19:31:58.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.949+0000 7f8ac5d1c640 1 librados: starting objecter 2024-03-25T19:31:58.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.949+0000 7f8ac5d1c640 1 Processor -- start 2024-03-25T19:31:58.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.949+0000 7f8ac5d1c640 1 -- start start 2024-03-25T19:31:58.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.949+0000 7f8ac5d1c640 1 librados: setting wanted keys 2024-03-25T19:31:58.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.949+0000 7f8ac5d1c640 1 librados: calling monclient init 2024-03-25T19:31:58.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.949+0000 7f8ac5d1c640 10 monclient: init 2024-03-25T19:31:58.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.950+0000 7f8ac5d1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:58.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.950+0000 7f8ac5d1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:58.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.950+0000 7f8ac5d1c640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac00a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:31:58.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.950+0000 7f8ac5d1c640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:31:58.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.950+0000 7f8ac5d1c640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac01430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:31:58.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.950+0000 7f8ac5d1c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac0138730 con 0x7f8ac01430c0 2024-03-25T19:31:58.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.950+0000 7f8ac5d1c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac0138a00 con 0x7f8ac013f710 2024-03-25T19:31:58.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.950+0000 7f8ac5d1c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac00b1790 con 0x7f8ac00a98f0 2024-03-25T19:31:58.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.950+0000 7f8ac5d1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:58.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.950+0000 7f8ac5d1c640 10 monclient(hunting): authenticate will time out at 6167.131377s 2024-03-25T19:31:58.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.957+0000 7f8abf7fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f8ac00a98f0 0x7f8ac01379d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37496/0 (socket says 172.21.15.52:37496) 2024-03-25T19:31:58.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.957+0000 7f8abf7fe640 1 -- 172.21.15.52:0/560088664 learned_addr learned my addr 172.21.15.52:0/560088664 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:31:58.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.958+0000 7f8aa7fff640 10 client.?.objecter ms_handle_connect 0x7f8ac00a98f0 2024-03-25T19:31:58.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.958+0000 7f8aa7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:58.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.959+0000 7f8aa7fff640 10 client.?.objecter ms_handle_connect 0x7f8ac01430c0 2024-03-25T19:31:58.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.959+0000 7f8aa7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:58.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.959+0000 7f8aa7fff640 10 client.?.objecter ms_handle_connect 0x7f8ac013f710 2024-03-25T19:31:58.964 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.959+0000 7f8aa7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:58.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.966+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487593421 0 0) 0x7f8ac0138730 con 0x7f8ac01430c0 2024-03-25T19:31:58.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.966+0000 7f8aa7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.966+0000 7f8aa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.966+0000 7f8aa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.966+0000 7f8aa7fff640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:31:58.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.966+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aac004eb0 con 0x7f8ac01430c0 2024-03-25T19:31:58.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.969+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715909708 0 0) 0x7f8ac00b1790 con 0x7f8ac00a98f0 2024-03-25T19:31:58.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.969+0000 7f8aa7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.969+0000 7f8aa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.969+0000 7f8aa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.969+0000 7f8aa7fff640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:31:58.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.969+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac0138730 con 0x7f8ac00a98f0 2024-03-25T19:31:58.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.973+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219388161 0 0) 0x7f8ac0138a00 con 0x7f8ac013f710 2024-03-25T19:31:58.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.973+0000 7f8aa7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.973+0000 7f8aa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.973+0000 7f8aa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.973+0000 7f8aa7fff640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:31:58.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.973+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac00b1790 con 0x7f8ac013f710 2024-03-25T19:31:58.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.980+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3374460778 0 0) 0x7f8aac004eb0 con 0x7f8ac01430c0 2024-03-25T19:31:58.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.980+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ac0138a00 con 0x7f8ac01430c0 2024-03-25T19:31:58.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.982+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2785553868 0 0) 0x7f8ac00b1790 con 0x7f8ac013f710 2024-03-25T19:31:58.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.982+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8aac004eb0 con 0x7f8ac013f710 2024-03-25T19:31:58.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.982+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8ab4001b00 con 0x7f8ac013f710 2024-03-25T19:31:58.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.982+0000 7f8aa7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.982+0000 7f8aa7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:58.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.982+0000 7f8aa7fff640 10 monclient(hunting): dump: 2024-03-25T19:31:58.983 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:58.983 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:58.983 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:58.984 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:58.984 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.984 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:58.984 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:58.984 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:58.984 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:58.984 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:58.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.982+0000 7f8aa7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:58.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8ab8001c10 con 0x7f8ac01430c0 2024-03-25T19:31:58.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:58.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 10 monclient(hunting): dump: 2024-03-25T19:31:58.990 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:58.990 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:58.990 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:58.990 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:58.990 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.990 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:58.990 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:58.990 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 306833904 0 0) 0x7f8ac0138a00 con 0x7f8ac01430c0 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 >> v1:172.21.15.52:6790/0 conn(0x7f8ac00a98f0 legacy=0x7f8ac01379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 >> v1:172.21.15.179:6789/0 conn(0x7f8ac013f710 legacy=0x7f8ac0134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 1 monclient: found mon.a 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac00b12e0 con 0x7f8ac01430c0 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8aa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8ac5d1c640 5 monclient: authenticate success, global_id 4425 2024-03-25T19:31:58.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8ac5d1c640 10 monclient: _renew_subs 2024-03-25T19:31:58.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8ac5d1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:58.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/560088664 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac00b1530 con 0x7f8ac01430c0 2024-03-25T19:31:58.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8ac5d1c640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:58.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8ac5d1c640 10 monclient: _renew_subs 2024-03-25T19:31:58.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8ac5d1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:58.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/560088664 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac0134900 con 0x7f8ac01430c0 2024-03-25T19:31:58.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.988+0000 7f8ac5d1c640 1 librados: init done 2024-03-25T19:31:58.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.989+0000 7f8ac5d1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:58.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.989+0000 7f8ac5d1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:58.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.989+0000 7f8ac5d1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:58.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:58.989+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/560088664 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac0138b90 con 0x7f8ac01430c0 2024-03-25T19:31:59.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.305+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ab80020e0 con 0x7f8ac01430c0 2024-03-25T19:31:59.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.305+0000 7f8aa7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:59.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.305+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8ab8003b30 con 0x7f8ac01430c0 2024-03-25T19:31:59.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.305+0000 7f8aa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:59.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.305+0000 7f8aa7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:59.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.305+0000 7f8aa7fff640 10 monclient: dump: 2024-03-25T19:31:59.306 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:31:59.306 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:31:59.306 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:59.306 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:31:59.306 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:59.307 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:31:59.307 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:31:59.307 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:31:59.307 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:31:59.307 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:31:59.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.305+0000 7f8abdffb640 4 set_mon_vals no callback set 2024-03-25T19:31:59.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.306+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f8ab8003d50 con 0x7f8ac01430c0 2024-03-25T19:31:59.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.306+0000 7f8abf7fe640 10 monclient: get_auth_request con 0x7f8aac06dff0 auth_method 0 2024-03-25T19:31:59.308 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.307+0000 7f8aa7fff640 10 client.4425.objecter ms_handle_connect 0x7f8aac06dff0 2024-03-25T19:31:59.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.313+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f8ab8053910 con 0x7f8ac01430c0 2024-03-25T19:31:59.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.313+0000 7f8aa7fff640 10 client.4425.objecter ms_dispatch 0x7f8ac0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:59.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.313+0000 7f8aa7fff640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:59.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.313+0000 7f8aa7fff640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:59.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.313+0000 7f8aa7fff640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-25T19:31:59.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.313+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ab804f250 con 0x7f8ac01430c0 2024-03-25T19:31:59.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.313+0000 7f8aa7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:59.314 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.313+0000 7f8aa7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:59.578 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.578+0000 7f8ac5d1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:59.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.578+0000 7f8ac5d1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:59.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.578+0000 7f8ac5d1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:31:59.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.578+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/560088664 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ac00627b0 con 0x7f8ac01430c0 2024-03-25T19:31:59.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.613+0000 7f8aa7fff640 1 -- 172.21.15.52:0/560088664 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f8ab8051350 con 0x7f8ac01430c0 2024-03-25T19:31:59.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.613+0000 7f8aa7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:59.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.613+0000 7f8aa7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:59.614 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:31:59.614 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:59.617 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 10 librados: watch_flush enter 2024-03-25T19:31:59.617 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 10 librados: watch_flush exit 2024-03-25T19:31:59.617 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-25T19:31:59.617 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 10 client.4425.objecter successfully canceled tick 2024-03-25T19:31:59.617 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/560088664 >> v1:172.21.15.52:6800/3967427147 conn(0x7f8aac06dff0 legacy=0x7f8aac0704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:59.617 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 10 monclient: shutdown 2024-03-25T19:31:59.617 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:59.617 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/560088664 >> v1:172.21.15.52:6789/0 conn(0x7f8ac01430c0 legacy=0x7f8ac01da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:59.617 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/560088664 shutdown_connections 2024-03-25T19:31:59.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/560088664 >> 172.21.15.52:0/560088664 conn(0x7f8ac00b7ea0 msgr2=0x7f8ac0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:59.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/560088664 shutdown_connections 2024-03-25T19:31:59.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 1 -- 172.21.15.52:0/560088664 wait complete. 2024-03-25T19:31:59.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:31:59.616+0000 7f8ac5d1c640 1 librados: shutdown 2024-03-25T19:31:59.628 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:06.436 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:06.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.555+0000 7f70ab3fb640 10 monclient: get_monmap_and_config 2024-03-25T19:32:06.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.555+0000 7f70ab3fb640 10 monclient: build_initial_monmap 2024-03-25T19:32:06.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.555+0000 7f70ab3fb640 10 monclient: monmap: 2024-03-25T19:32:06.556 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:06.556 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:06.557 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:32:06.557178+0000 2024-03-25T19:32:06.557 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:32:06.557178+0000 2024-03-25T19:32:06.557 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:06.557 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:06.557 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:06.557 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:32:06.557 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:32:06.557 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:06.557 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.556+0000 7f70ab3fb640 1 Processor -- start 2024-03-25T19:32:06.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.556+0000 7f70ab3fb640 1 -- start start 2024-03-25T19:32:06.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.556+0000 7f70ab3fb640 10 monclient: init 2024-03-25T19:32:06.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.557+0000 7f70ab3fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:06.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.557+0000 7f70ab3fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:32:06.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.557+0000 7f70ab3fb640 10 monclient(hunting): picked mon.noname-a con 0x7f70a40a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:06.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.557+0000 7f70ab3fb640 10 monclient(hunting): picked mon.noname-b con 0x7f70a413f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:06.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.557+0000 7f70ab3fb640 10 monclient(hunting): picked mon.noname-c con 0x7f70a41430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:06.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.557+0000 7f70ab3fb640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a4142520 con 0x7f70a40a98f0 2024-03-25T19:32:06.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.557+0000 7f70ab3fb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a40b28a0 con 0x7f70a41430c0 2024-03-25T19:32:06.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.557+0000 7f70ab3fb640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a40b2a30 con 0x7f70a413f710 2024-03-25T19:32:06.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.557+0000 7f70ab3fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:06.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.557+0000 7f70ab3fb640 10 monclient(hunting): authenticate will time out at 6174.738271s 2024-03-25T19:32:06.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.569+0000 7f70a9998640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f70a41430c0 0x7f70a414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:34548/0 (socket says 172.21.15.52:34548) 2024-03-25T19:32:06.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.569+0000 7f70a9998640 1 -- 172.21.15.52:0/1559596779 learned_addr learned my addr 172.21.15.52:0/1559596779 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:06.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.581+0000 7f709bfff640 1 -- 172.21.15.52:0/1559596779 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755682366 0 0) 0x7f70a40b28a0 con 0x7f70a41430c0 2024-03-25T19:32:06.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.581+0000 7f709bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:06.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.581+0000 7f709bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:06.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.581+0000 7f709bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:06.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.581+0000 7f709bfff640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:32:06.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.581+0000 7f709bfff640 1 -- 172.21.15.52:0/1559596779 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7090003520 con 0x7f70a41430c0 2024-03-25T19:32:06.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.585+0000 7f709bfff640 1 -- 172.21.15.52:0/1559596779 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411743790 0 0) 0x7f70a4142520 con 0x7f70a40a98f0 2024-03-25T19:32:06.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.585+0000 7f709bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:06.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.585+0000 7f709bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:06.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.585+0000 7f709bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:06.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.585+0000 7f709bfff640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:32:06.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.585+0000 7f709bfff640 1 -- 172.21.15.52:0/1559596779 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a40b28a0 con 0x7f70a40a98f0 2024-03-25T19:32:06.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 1 -- 172.21.15.52:0/1559596779 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2483380767 0 0) 0x7f7090003520 con 0x7f70a41430c0 2024-03-25T19:32:06.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 1 -- 172.21.15.52:0/1559596779 >> v1:172.21.15.52:6790/0 conn(0x7f70a413f710 legacy=0x7f70a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 1 -- 172.21.15.52:0/1559596779 >> v1:172.21.15.52:6789/0 conn(0x7f70a40a98f0 legacy=0x7f70a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:06.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 1 monclient: found mon.noname-c 2024-03-25T19:32:06.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:06.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:32:06.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 1 -- 172.21.15.52:0/1559596779 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a40b44c0 con 0x7f70a41430c0 2024-03-25T19:32:06.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 10 monclient: _finish_auth 0 2024-03-25T19:32:06.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:06.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 1 -- 172.21.15.52:0/1559596779 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f709c001a10 con 0x7f70a41430c0 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f70ab3fb640 5 monclient: authenticate success, global_id 4435 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f70ab3fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f709bfff640 10 monclient: dump: 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:06.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.595+0000 7f70ab3fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:06.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.601+0000 7f709bfff640 1 -- 172.21.15.52:0/1559596779 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f709c001b70 con 0x7f70a41430c0 2024-03-25T19:32:06.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.601+0000 7f709bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:06.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.601+0000 7f709bfff640 1 -- 172.21.15.52:0/1559596779 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f709c001f80 con 0x7f70a41430c0 2024-03-25T19:32:06.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.601+0000 7f709bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:06.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.601+0000 7f709bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.601+0000 7f709bfff640 10 monclient: dump: 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.601+0000 7f70ab3fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:06.603 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.601+0000 7f70ab3fb640 4 set_mon_vals no callback set 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.602+0000 7f70ab3fb640 10 monclient: shutdown 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.602+0000 7f70ab3fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.602+0000 7f70ab3fb640 1 -- 172.21.15.52:0/1559596779 >> v1:172.21.15.179:6789/0 conn(0x7f70a41430c0 legacy=0x7f70a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.602+0000 7f70ab3fb640 1 -- 172.21.15.52:0/1559596779 shutdown_connections 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.602+0000 7f70ab3fb640 1 -- 172.21.15.52:0/1559596779 >> 172.21.15.52:0/1559596779 conn(0x7f70a40b7ea0 msgr2=0x7f70a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.603+0000 7f70ab3fb640 1 -- 172.21.15.52:0/1559596779 shutdown_connections 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.603+0000 7f70ab3fb640 1 -- 172.21.15.52:0/1559596779 wait complete. 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.603+0000 7f70ab3fb640 10 monclient: build_initial_monmap 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.603+0000 7f70ab3fb640 10 monclient: monmap: 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:06.605 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.604+0000 7f70ab3fb640 1 librados: starting msgr at 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.604+0000 7f70ab3fb640 1 librados: starting objecter 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.604+0000 7f70ab3fb640 1 Processor -- start 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.604+0000 7f70ab3fb640 1 -- start start 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.604+0000 7f70ab3fb640 1 librados: setting wanted keys 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.604+0000 7f70ab3fb640 1 librados: calling monclient init 2024-03-25T19:32:06.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.604+0000 7f70ab3fb640 10 monclient: init 2024-03-25T19:32:06.607 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.605+0000 7f70ab3fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:06.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.605+0000 7f70ab3fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:06.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.605+0000 7f70ab3fb640 10 monclient(hunting): picked mon.noname-c con 0x7f70a40a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:06.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.605+0000 7f70ab3fb640 10 monclient(hunting): picked mon.noname-a con 0x7f70a413f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:06.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.605+0000 7f70ab3fb640 10 monclient(hunting): picked mon.noname-b con 0x7f70a41430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:06.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.605+0000 7f70ab3fb640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a4135360 con 0x7f70a413f710 2024-03-25T19:32:06.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.605+0000 7f70ab3fb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a4139a50 con 0x7f70a41430c0 2024-03-25T19:32:06.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.605+0000 7f70ab3fb640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a4136b10 con 0x7f70a40a98f0 2024-03-25T19:32:06.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.605+0000 7f70ab3fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:06.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.605+0000 7f70ab3fb640 10 monclient(hunting): authenticate will time out at 6174.786233s 2024-03-25T19:32:06.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.609+0000 7f70a9998640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f70a41430c0 0x7f70a41e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:34586/0 (socket says 172.21.15.52:34586) 2024-03-25T19:32:06.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.609+0000 7f70a9998640 1 -- 172.21.15.52:0/910135929 learned_addr learned my addr 172.21.15.52:0/910135929 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:06.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.609+0000 7f7099ffb640 10 client.?.objecter ms_handle_connect 0x7f70a41430c0 2024-03-25T19:32:06.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.609+0000 7f7099ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:06.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.614+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015414248 0 0) 0x7f70a4139a50 con 0x7f70a41430c0 2024-03-25T19:32:06.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.614+0000 7f7099ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:06.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.614+0000 7f7099ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:06.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.614+0000 7f7099ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:06.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.614+0000 7f7099ffb640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:32:06.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.614+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7088004eb0 con 0x7f70a41430c0 2024-03-25T19:32:06.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.619+0000 7f7099ffb640 10 client.?.objecter ms_handle_connect 0x7f70a40a98f0 2024-03-25T19:32:06.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.619+0000 7f7099ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:06.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.619+0000 7f7099ffb640 10 client.?.objecter ms_handle_connect 0x7f70a413f710 2024-03-25T19:32:06.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.619+0000 7f7099ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:06.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.624+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4179682724 0 0) 0x7f7088004eb0 con 0x7f70a41430c0 2024-03-25T19:32:06.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.624+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70a4139a50 con 0x7f70a41430c0 2024-03-25T19:32:06.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.624+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f709c001ec0 con 0x7f70a41430c0 2024-03-25T19:32:06.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.624+0000 7f7099ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:06.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.624+0000 7f7099ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:06.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.624+0000 7f7099ffb640 10 monclient(hunting): dump: 2024-03-25T19:32:06.625 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:06.625 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:06.626 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:06.626 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:06.626 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:06.626 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:06.626 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:06.626 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:06.626 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:06.626 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:06.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.624+0000 7f7099ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:06.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.633+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910529272 0 0) 0x7f70a4136b10 con 0x7f70a40a98f0 2024-03-25T19:32:06.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.633+0000 7f7099ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:06.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.633+0000 7f7099ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:06.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.633+0000 7f7099ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:06.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.633+0000 7f7099ffb640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:32:06.634 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.633+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7088004eb0 con 0x7f70a40a98f0 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 319041094 0 0) 0x7f70a4139a50 con 0x7f70a41430c0 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 >> v1:172.21.15.52:6790/0 conn(0x7f70a40a98f0 legacy=0x7f70a41374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 >> v1:172.21.15.52:6789/0 conn(0x7f70a413f710 legacy=0x7f70a40b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f7099ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f7099ffb640 1 monclient: found mon.b 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f7099ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f7099ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a4136df0 con 0x7f70a41430c0 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f7099ffb640 10 monclient: _finish_auth 0 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f7099ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f7099ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f70ab3fb640 5 monclient: authenticate success, global_id 4438 2024-03-25T19:32:06.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f70ab3fb640 10 monclient: _renew_subs 2024-03-25T19:32:06.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f70ab3fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:06.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f70ab3fb640 1 -- 172.21.15.52:0/910135929 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a41370d0 con 0x7f70a41430c0 2024-03-25T19:32:06.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f70ab3fb640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:06.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f70ab3fb640 10 monclient: _renew_subs 2024-03-25T19:32:06.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f70ab3fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:06.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f70ab3fb640 1 -- 172.21.15.52:0/910135929 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a4135fb0 con 0x7f70a41430c0 2024-03-25T19:32:06.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.637+0000 7f70ab3fb640 1 librados: init done 2024-03-25T19:32:06.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.638+0000 7f70ab3fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:06.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.638+0000 7f70ab3fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:06.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.638+0000 7f70ab3fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:06.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.639+0000 7f70ab3fb640 1 -- 172.21.15.52:0/910135929 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a40627b0 con 0x7f70a41430c0 2024-03-25T19:32:06.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.916+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f709c004e10 con 0x7f70a41430c0 2024-03-25T19:32:06.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.916+0000 7f7099ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:06.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.916+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f709c005120 con 0x7f70a41430c0 2024-03-25T19:32:06.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.916+0000 7f7099ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:06.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.916+0000 7f7099ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:06.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.916+0000 7f7099ffb640 10 monclient: dump: 2024-03-25T19:32:06.917 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:06.918 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:06.918 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:06.918 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:06.918 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:06.918 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:06.918 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:06.918 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:06.918 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:06.918 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:06.918 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.916+0000 7f709b7fe640 4 set_mon_vals no callback set 2024-03-25T19:32:06.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.917+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f709c01b0f0 con 0x7f70a41430c0 2024-03-25T19:32:06.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.918+0000 7f70a9197640 10 monclient: get_auth_request con 0x7f708806dd90 auth_method 0 2024-03-25T19:32:06.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.918+0000 7f7099ffb640 10 client.4438.objecter ms_handle_connect 0x7f708806dd90 2024-03-25T19:32:06.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.922+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f709c054b60 con 0x7f70a41430c0 2024-03-25T19:32:06.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.922+0000 7f7099ffb640 10 client.4438.objecter ms_dispatch 0x7f70a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:06.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.922+0000 7f7099ffb640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:06.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.922+0000 7f7099ffb640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:06.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.922+0000 7f7099ffb640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-25T19:32:06.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.924+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f709c050670 con 0x7f70a41430c0 2024-03-25T19:32:06.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.924+0000 7f7099ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:06.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:06.924+0000 7f7099ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:06.951 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:05:00.549 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:07.010 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:05:00.606 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:07.036 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:05:00.635 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:07.061 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:05:00.603 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:07.070 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:05:00.612 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:07.093 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:05:00.635 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:07.097 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:05:00.696 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:07.103 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:05:00.702 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:07.143 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:05:00.742 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:07.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.162+0000 7f70ab3fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:07.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.162+0000 7f70ab3fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:07.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.162+0000 7f70ab3fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:07.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.162+0000 7f70ab3fb640 1 -- 172.21.15.52:0/910135929 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70a413dcf0 con 0x7f70a41430c0 2024-03-25T19:32:07.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.210+0000 7f7099ffb640 1 -- 172.21.15.52:0/910135929 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f709c0525a0 con 0x7f70a41430c0 2024-03-25T19:32:07.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.210+0000 7f7099ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:07.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.210+0000 7f7099ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:07.211 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:32:07.211 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:07.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.212+0000 7f70ab3fb640 10 librados: watch_flush enter 2024-03-25T19:32:07.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.212+0000 7f70ab3fb640 10 librados: watch_flush exit 2024-03-25T19:32:07.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.212+0000 7f70ab3fb640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-25T19:32:07.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.212+0000 7f70ab3fb640 10 client.4438.objecter successfully canceled tick 2024-03-25T19:32:07.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.212+0000 7f70ab3fb640 1 -- 172.21.15.52:0/910135929 >> v1:172.21.15.52:6800/3967427147 conn(0x7f708806dd90 legacy=0x7f7088070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:07.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.212+0000 7f70ab3fb640 10 monclient: shutdown 2024-03-25T19:32:07.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.212+0000 7f70ab3fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:07.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.212+0000 7f70ab3fb640 1 -- 172.21.15.52:0/910135929 >> v1:172.21.15.179:6789/0 conn(0x7f70a41430c0 legacy=0x7f70a41e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:07.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.213+0000 7f70ab3fb640 1 -- 172.21.15.52:0/910135929 shutdown_connections 2024-03-25T19:32:07.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.213+0000 7f70ab3fb640 1 -- 172.21.15.52:0/910135929 >> 172.21.15.52:0/910135929 conn(0x7f70a40b7ea0 msgr2=0x7f70a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:07.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.213+0000 7f70ab3fb640 1 -- 172.21.15.52:0/910135929 shutdown_connections 2024-03-25T19:32:07.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.213+0000 7f70ab3fb640 1 -- 172.21.15.52:0/910135929 wait complete. 2024-03-25T19:32:07.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:07.213+0000 7f70ab3fb640 1 librados: shutdown 2024-03-25T19:32:07.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:14.030 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:14.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.143+0000 7ff039250640 10 monclient: get_monmap_and_config 2024-03-25T19:32:14.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.144+0000 7ff039250640 10 monclient: build_initial_monmap 2024-03-25T19:32:14.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.144+0000 7ff039250640 10 monclient: monmap: 2024-03-25T19:32:14.144 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:14.145 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:14.145 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:32:14.145577+0000 2024-03-25T19:32:14.145 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:32:14.145577+0000 2024-03-25T19:32:14.145 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:14.145 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:14.145 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:14.145 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:32:14.145 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:32:14.145 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:14.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.144+0000 7ff039250640 1 Processor -- start 2024-03-25T19:32:14.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.144+0000 7ff039250640 1 -- start start 2024-03-25T19:32:14.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.144+0000 7ff039250640 10 monclient: init 2024-03-25T19:32:14.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.145+0000 7ff039250640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:14.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.145+0000 7ff039250640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:32:14.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.145+0000 7ff039250640 10 monclient(hunting): picked mon.noname-c con 0x7ff0340b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:14.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.145+0000 7ff039250640 10 monclient(hunting): picked mon.noname-a con 0x7ff0340b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:14.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.145+0000 7ff039250640 10 monclient(hunting): picked mon.noname-b con 0x7ff03414be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:14.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.145+0000 7ff039250640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0340af940 con 0x7ff0340b1730 2024-03-25T19:32:14.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.145+0000 7ff039250640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0340b0a70 con 0x7ff0340b4d60 2024-03-25T19:32:14.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.145+0000 7ff039250640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0340b0d50 con 0x7ff03414be90 2024-03-25T19:32:14.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.145+0000 7ff039250640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:14.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.145+0000 7ff039250640 10 monclient(hunting): authenticate will time out at 6182.326408s 2024-03-25T19:32:14.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.165+0000 7ff032d9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7ff0340b4d60 0x7ff03413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:38962/0 (socket says 172.21.15.52:38962) 2024-03-25T19:32:14.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.165+0000 7ff032d9d640 1 -- 172.21.15.52:0/3715772923 learned_addr learned my addr 172.21.15.52:0/3715772923 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:14.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.177+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660832000 0 0) 0x7ff0340b0a70 con 0x7ff0340b4d60 2024-03-25T19:32:14.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.177+0000 7ff031d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:32:14.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.177+0000 7ff031d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:14.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.177+0000 7ff031d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:14.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.177+0000 7ff031d9b640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:32:14.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.177+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff01c003520 con 0x7ff0340b4d60 2024-03-25T19:32:14.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.178+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941290291 0 0) 0x7ff0340af940 con 0x7ff0340b1730 2024-03-25T19:32:14.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.178+0000 7ff031d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:32:14.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.178+0000 7ff031d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:14.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.178+0000 7ff031d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:14.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.178+0000 7ff031d9b640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:32:14.179 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.178+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0340b0a70 con 0x7ff0340b1730 2024-03-25T19:32:14.189 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.189+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955712553 0 0) 0x7ff0340b0d50 con 0x7ff03414be90 2024-03-25T19:32:14.189 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.189+0000 7ff031d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:32:14.189 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.189+0000 7ff031d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:14.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.189+0000 7ff031d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:14.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.189+0000 7ff031d9b640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:32:14.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.189+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0340af940 con 0x7ff03414be90 2024-03-25T19:32:14.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 653106785 0 0) 0x7ff01c003520 con 0x7ff0340b4d60 2024-03-25T19:32:14.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 >> v1:172.21.15.52:6790/0 conn(0x7ff03414be90 legacy=0x7ff03414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 >> v1:172.21.15.52:6789/0 conn(0x7ff0340b1730 legacy=0x7ff0340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:32:14.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 1 monclient: found mon.noname-c 2024-03-25T19:32:14.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0340b1030 con 0x7ff0340b4d60 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 10 monclient: _finish_auth 0 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff024001690 con 0x7ff0340b4d60 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff039250640 5 monclient: authenticate success, global_id 4441 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff039250640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff031d9b640 10 monclient: dump: 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.195 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.196 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:14.196 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:14.196 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:14.196 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:14.196 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:14.196 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:14.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.193+0000 7ff039250640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:14.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.217+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0240017f0 con 0x7ff0340b4d60 2024-03-25T19:32:14.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.217+0000 7ff031d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:14.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.217+0000 7ff031d9b640 1 -- 172.21.15.52:0/3715772923 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff024001c00 con 0x7ff0340b4d60 2024-03-25T19:32:14.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.217+0000 7ff039250640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:14.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.217+0000 7ff039250640 4 set_mon_vals no callback set 2024-03-25T19:32:14.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.217+0000 7ff031d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:14.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.217+0000 7ff031d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:14.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.217+0000 7ff031d9b640 10 monclient: dump: 2024-03-25T19:32:14.218 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:14.219 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:14.219 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.219 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.219 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:14.219 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:14.219 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:14.219 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:14.219 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:14.219 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:14.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.218+0000 7ff039250640 10 monclient: shutdown 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.218+0000 7ff039250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.218+0000 7ff039250640 1 -- 172.21.15.52:0/3715772923 >> v1:172.21.15.179:6789/0 conn(0x7ff0340b4d60 legacy=0x7ff03413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.218+0000 7ff039250640 1 -- 172.21.15.52:0/3715772923 shutdown_connections 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.218+0000 7ff039250640 1 -- 172.21.15.52:0/3715772923 >> 172.21.15.52:0/3715772923 conn(0x7ff0340617a0 msgr2=0x7ff034061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.218+0000 7ff039250640 1 -- 172.21.15.52:0/3715772923 shutdown_connections 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.218+0000 7ff039250640 1 -- 172.21.15.52:0/3715772923 wait complete. 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.219+0000 7ff039250640 10 monclient: build_initial_monmap 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.219+0000 7ff039250640 10 monclient: monmap: 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:14.221 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:32:14.222 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:32:14.222 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:14.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.220+0000 7ff039250640 1 librados: starting msgr at 2024-03-25T19:32:14.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.220+0000 7ff039250640 1 librados: starting objecter 2024-03-25T19:32:14.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.220+0000 7ff039250640 1 Processor -- start 2024-03-25T19:32:14.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.220+0000 7ff039250640 1 -- start start 2024-03-25T19:32:14.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.220+0000 7ff039250640 1 librados: setting wanted keys 2024-03-25T19:32:14.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.220+0000 7ff039250640 1 librados: calling monclient init 2024-03-25T19:32:14.222 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.220+0000 7ff039250640 10 monclient: init 2024-03-25T19:32:14.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.221+0000 7ff039250640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:14.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.221+0000 7ff039250640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:14.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.221+0000 7ff039250640 10 monclient(hunting): picked mon.noname-c con 0x7ff0340b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:14.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.221+0000 7ff039250640 10 monclient(hunting): picked mon.noname-b con 0x7ff0340b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:14.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.221+0000 7ff039250640 10 monclient(hunting): picked mon.noname-a con 0x7ff03414be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:14.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.221+0000 7ff039250640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff034140d00 con 0x7ff03414be90 2024-03-25T19:32:14.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.221+0000 7ff039250640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff034141cc0 con 0x7ff0340b4d60 2024-03-25T19:32:14.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.221+0000 7ff039250640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0341d8470 con 0x7ff0340b1730 2024-03-25T19:32:14.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.221+0000 7ff039250640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:14.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.221+0000 7ff039250640 10 monclient(hunting): authenticate will time out at 6182.402481s 2024-03-25T19:32:14.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.228+0000 7ff032d9d640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7ff0340b1730 0x7ff0341405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:60870/0 (socket says 172.21.15.52:60870) 2024-03-25T19:32:14.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.228+0000 7ff032d9d640 1 -- 172.21.15.52:0/2905364039 learned_addr learned my addr 172.21.15.52:0/2905364039 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:14.230 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.230+0000 7ff01b7fe640 10 client.?.objecter ms_handle_connect 0x7ff0340b1730 2024-03-25T19:32:14.230 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.230+0000 7ff01b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:14.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.231+0000 7ff01b7fe640 10 client.?.objecter ms_handle_connect 0x7ff0340b4d60 2024-03-25T19:32:14.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.231+0000 7ff01b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:14.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.242+0000 7ff01b7fe640 10 client.?.objecter ms_handle_connect 0x7ff03414be90 2024-03-25T19:32:14.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.242+0000 7ff01b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:14.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.247+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568668310 0 0) 0x7ff0341d8470 con 0x7ff0340b1730 2024-03-25T19:32:14.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.247+0000 7ff01b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:32:14.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.247+0000 7ff01b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:14.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.247+0000 7ff01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:14.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.247+0000 7ff01b7fe640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:32:14.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.247+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff020004eb0 con 0x7ff0340b1730 2024-03-25T19:32:14.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.253+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266471767 0 0) 0x7ff034141cc0 con 0x7ff0340b4d60 2024-03-25T19:32:14.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.253+0000 7ff01b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:32:14.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.253+0000 7ff01b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:14.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.253+0000 7ff01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:14.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.253+0000 7ff01b7fe640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:32:14.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.253+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0341d8470 con 0x7ff0340b4d60 2024-03-25T19:32:14.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.260+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728129673 0 0) 0x7ff034140d00 con 0x7ff03414be90 2024-03-25T19:32:14.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.260+0000 7ff01b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:32:14.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.260+0000 7ff01b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:14.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.260+0000 7ff01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:14.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.260+0000 7ff01b7fe640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:32:14.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.260+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff034141cc0 con 0x7ff03414be90 2024-03-25T19:32:14.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.271+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4033494260 0 0) 0x7ff020004eb0 con 0x7ff0340b1730 2024-03-25T19:32:14.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.271+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff034140d00 con 0x7ff0340b1730 2024-03-25T19:32:14.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.271+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff024001bc0 con 0x7ff0340b1730 2024-03-25T19:32:14.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.271+0000 7ff01b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:14.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.271+0000 7ff01b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:14.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.271+0000 7ff01b7fe640 10 monclient(hunting): dump: 2024-03-25T19:32:14.272 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:14.273 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:14.273 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.273 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.273 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:14.273 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:14.273 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:14.273 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:14.273 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:14.273 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:14.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.271+0000 7ff01b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:14.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.272+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 326543253 0 0) 0x7ff034141cc0 con 0x7ff03414be90 2024-03-25T19:32:14.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.272+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff020004eb0 con 0x7ff03414be90 2024-03-25T19:32:14.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.272+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff02c001bd0 con 0x7ff03414be90 2024-03-25T19:32:14.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.272+0000 7ff01b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:14.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.272+0000 7ff01b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:14.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.272+0000 7ff01b7fe640 10 monclient(hunting): dump: 2024-03-25T19:32:14.274 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:14.274 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:14.274 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.274 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.274 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:14.275 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:14.275 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:14.275 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:14.275 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:14.275 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:14.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.276+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 456288623 0 0) 0x7ff0341d8470 con 0x7ff0340b4d60 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.276+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff034141cc0 con 0x7ff0340b4d60 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.276+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff028001970 con 0x7ff0340b4d60 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.276+0000 7ff01b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.276+0000 7ff01b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.276+0000 7ff01b7fe640 10 monclient(hunting): dump: 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:14.277 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:14.278 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:14.286 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1028944776 0 0) 0x7ff020004eb0 con 0x7ff03414be90 2024-03-25T19:32:14.286 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 >> v1:172.21.15.52:6790/0 conn(0x7ff0340b1730 legacy=0x7ff0341405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.286 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 >> v1:172.21.15.179:6789/0 conn(0x7ff0340b4d60 legacy=0x7ff0341d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.286 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff01b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:32:14.286 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff01b7fe640 1 monclient: found mon.a 2024-03-25T19:32:14.286 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff01b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:14.286 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff01b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0341d8600 con 0x7ff03414be90 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff01b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff01b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff01b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff039250640 5 monclient: authenticate success, global_id 4437 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff039250640 10 monclient: _renew_subs 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff039250640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff039250640 1 -- 172.21.15.52:0/2905364039 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03413b3c0 con 0x7ff03414be90 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff039250640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff039250640 10 monclient: _renew_subs 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff039250640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff039250640 1 -- 172.21.15.52:0/2905364039 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0341425a0 con 0x7ff03414be90 2024-03-25T19:32:14.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.285+0000 7ff039250640 1 librados: init done 2024-03-25T19:32:14.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.286+0000 7ff039250640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:14.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.287+0000 7ff039250640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:14.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.287+0000 7ff039250640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:14.288 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.287+0000 7ff039250640 1 -- 172.21.15.52:0/2905364039 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0340b2270 con 0x7ff03414be90 2024-03-25T19:32:14.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.605+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff02c001fe0 con 0x7ff03414be90 2024-03-25T19:32:14.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.605+0000 7ff01b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:14.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.605+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff02c003b30 con 0x7ff03414be90 2024-03-25T19:32:14.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.605+0000 7ff01b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:14.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.605+0000 7ff01b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:14.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.605+0000 7ff01b7fe640 10 monclient: dump: 2024-03-25T19:32:14.606 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:14.606 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:14.607 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.607 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:14.607 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:14.607 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:14.607 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:14.607 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:14.607 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:14.607 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:14.607 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.605+0000 7ff03159a640 4 set_mon_vals no callback set 2024-03-25T19:32:14.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.606+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7ff02c019b00 con 0x7ff03414be90 2024-03-25T19:32:14.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.606+0000 7ff032d9d640 10 monclient: get_auth_request con 0x7ff02006e110 auth_method 0 2024-03-25T19:32:14.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.606+0000 7ff01b7fe640 10 client.4437.objecter ms_handle_connect 0x7ff02006e110 2024-03-25T19:32:14.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.612+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7ff02c053770 con 0x7ff03414be90 2024-03-25T19:32:14.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.612+0000 7ff01b7fe640 10 client.4437.objecter ms_dispatch 0x7ff034000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:14.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.612+0000 7ff01b7fe640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:14.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.612+0000 7ff01b7fe640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:14.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.612+0000 7ff01b7fe640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-25T19:32:14.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.612+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff02c04f0b0 con 0x7ff03414be90 2024-03-25T19:32:14.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.612+0000 7ff01b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:14.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.612+0000 7ff01b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:14.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.858+0000 7ff039250640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:14.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.858+0000 7ff039250640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:14.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.858+0000 7ff039250640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:14.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.858+0000 7ff039250640 1 -- 172.21.15.52:0/2905364039 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0340627b0 con 0x7ff03414be90 2024-03-25T19:32:14.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.892+0000 7ff01b7fe640 1 -- 172.21.15.52:0/2905364039 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7ff02c0511b0 con 0x7ff03414be90 2024-03-25T19:32:14.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.892+0000 7ff01b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:14.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.892+0000 7ff01b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:14.893 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:32:14.893 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 10 librados: watch_flush enter 2024-03-25T19:32:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 10 librados: watch_flush exit 2024-03-25T19:32:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-25T19:32:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 10 client.4437.objecter successfully canceled tick 2024-03-25T19:32:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 1 -- 172.21.15.52:0/2905364039 >> v1:172.21.15.52:6800/3967427147 conn(0x7ff02006e110 legacy=0x7ff0200705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 10 monclient: shutdown 2024-03-25T19:32:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:14.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 1 -- 172.21.15.52:0/2905364039 >> v1:172.21.15.52:6789/0 conn(0x7ff03414be90 legacy=0x7ff034134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 1 -- 172.21.15.52:0/2905364039 shutdown_connections 2024-03-25T19:32:14.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 1 -- 172.21.15.52:0/2905364039 >> 172.21.15.52:0/2905364039 conn(0x7ff0340617a0 msgr2=0x7ff0340b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:14.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 1 -- 172.21.15.52:0/2905364039 shutdown_connections 2024-03-25T19:32:14.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 1 -- 172.21.15.52:0/2905364039 wait complete. 2024-03-25T19:32:14.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:14.894+0000 7ff039250640 1 librados: shutdown 2024-03-25T19:32:14.905 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:21.712 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:21.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.825+0000 7fc906a63640 10 monclient: get_monmap_and_config 2024-03-25T19:32:21.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.825+0000 7fc906a63640 10 monclient: build_initial_monmap 2024-03-25T19:32:21.827 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.825+0000 7fc906a63640 10 monclient: monmap: 2024-03-25T19:32:21.827 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:21.827 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:21.827 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:32:21.827527+0000 2024-03-25T19:32:21.827 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:32:21.827527+0000 2024-03-25T19:32:21.827 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:21.827 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:21.827 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:21.827 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:32:21.827 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:32:21.827 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:21.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.826+0000 7fc906a63640 1 Processor -- start 2024-03-25T19:32:21.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.826+0000 7fc906a63640 1 -- start start 2024-03-25T19:32:21.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.826+0000 7fc906a63640 10 monclient: init 2024-03-25T19:32:21.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.827+0000 7fc906a63640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:21.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.827+0000 7fc906a63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:32:21.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.827+0000 7fc906a63640 10 monclient(hunting): picked mon.noname-c con 0x7fc9000a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:21.828 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.827+0000 7fc906a63640 10 monclient(hunting): picked mon.noname-a con 0x7fc90013f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:21.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.827+0000 7fc906a63640 10 monclient(hunting): picked mon.noname-b con 0x7fc9001430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:21.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.827+0000 7fc906a63640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900142520 con 0x7fc90013f710 2024-03-25T19:32:21.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.827+0000 7fc906a63640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000b28a0 con 0x7fc9000a98f0 2024-03-25T19:32:21.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.827+0000 7fc906a63640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000b2a30 con 0x7fc9001430c0 2024-03-25T19:32:21.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.827+0000 7fc906a63640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:21.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.827+0000 7fc906a63640 10 monclient(hunting): authenticate will time out at 6190.008611s 2024-03-25T19:32:21.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.839+0000 7fc8ff7fe640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fc90013f710 0x7fc9001429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:53122/0 (socket says 172.21.15.52:53122) 2024-03-25T19:32:21.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.839+0000 7fc8ff7fe640 1 -- 172.21.15.52:0/1972135570 learned_addr learned my addr 172.21.15.52:0/1972135570 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:21.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.856+0000 7fc8feffd640 1 -- 172.21.15.52:0/1972135570 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780198362 0 0) 0x7fc9000b28a0 con 0x7fc9000a98f0 2024-03-25T19:32:21.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.856+0000 7fc8feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.856+0000 7fc8feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.856+0000 7fc8feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.856+0000 7fc8feffd640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:32:21.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.856+0000 7fc8feffd640 1 -- 172.21.15.52:0/1972135570 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e8003520 con 0x7fc9000a98f0 2024-03-25T19:32:21.862 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.862+0000 7fc8feffd640 1 -- 172.21.15.52:0/1972135570 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683905824 0 0) 0x7fc900142520 con 0x7fc90013f710 2024-03-25T19:32:21.862 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.862+0000 7fc8feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.863 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.862+0000 7fc8feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.863 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.862+0000 7fc8feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.863 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.862+0000 7fc8feffd640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:32:21.863 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.862+0000 7fc8feffd640 1 -- 172.21.15.52:0/1972135570 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9000b28a0 con 0x7fc90013f710 2024-03-25T19:32:21.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 1 -- 172.21.15.52:0/1972135570 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3847141093 0 0) 0x7fc8e8003520 con 0x7fc9000a98f0 2024-03-25T19:32:21.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 1 -- 172.21.15.52:0/1972135570 >> v1:172.21.15.52:6790/0 conn(0x7fc9001430c0 legacy=0x7fc90014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 1 -- 172.21.15.52:0/1972135570 >> v1:172.21.15.52:6789/0 conn(0x7fc90013f710 legacy=0x7fc9001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:21.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 1 monclient: found mon.noname-c 2024-03-25T19:32:21.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 1 -- 172.21.15.52:0/1972135570 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000b44c0 con 0x7fc9000a98f0 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 10 monclient: _finish_auth 0 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 1 -- 172.21.15.52:0/1972135570 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc8f0001690 con 0x7fc9000a98f0 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc8feffd640 10 monclient: dump: 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:21.866 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.867 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:21.867 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:21.867 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:21.867 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:21.867 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:21.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc906a63640 5 monclient: authenticate success, global_id 4447 2024-03-25T19:32:21.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.864+0000 7fc906a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:21.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.872+0000 7fc8feffd640 1 -- 172.21.15.52:0/1972135570 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8f00017f0 con 0x7fc9000a98f0 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.872+0000 7fc8feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.872+0000 7fc8feffd640 1 -- 172.21.15.52:0/1972135570 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc8f0001c00 con 0x7fc9000a98f0 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.872+0000 7fc8feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.872+0000 7fc8feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.872+0000 7fc8feffd640 10 monclient: dump: 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:21.873 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:21.874 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:21.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.872+0000 7fc906a63640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:21.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.872+0000 7fc906a63640 4 set_mon_vals no callback set 2024-03-25T19:32:21.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.873+0000 7fc906a63640 10 monclient: shutdown 2024-03-25T19:32:21.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.873+0000 7fc906a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:21.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.873+0000 7fc906a63640 1 -- 172.21.15.52:0/1972135570 >> v1:172.21.15.179:6789/0 conn(0x7fc9000a98f0 legacy=0x7fc9000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.873+0000 7fc906a63640 1 -- 172.21.15.52:0/1972135570 shutdown_connections 2024-03-25T19:32:21.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.873+0000 7fc906a63640 1 -- 172.21.15.52:0/1972135570 >> 172.21.15.52:0/1972135570 conn(0x7fc9000b7ea0 msgr2=0x7fc9000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:21.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.873+0000 7fc906a63640 1 -- 172.21.15.52:0/1972135570 shutdown_connections 2024-03-25T19:32:21.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.873+0000 7fc906a63640 1 -- 172.21.15.52:0/1972135570 wait complete. 2024-03-25T19:32:21.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.874+0000 7fc906a63640 10 monclient: build_initial_monmap 2024-03-25T19:32:21.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.874+0000 7fc906a63640 10 monclient: monmap: 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.874+0000 7fc906a63640 1 librados: starting msgr at 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.874+0000 7fc906a63640 1 librados: starting objecter 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.874+0000 7fc906a63640 1 Processor -- start 2024-03-25T19:32:21.877 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.874+0000 7fc906a63640 1 -- start start 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.874+0000 7fc906a63640 1 librados: setting wanted keys 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.874+0000 7fc906a63640 1 librados: calling monclient init 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.874+0000 7fc906a63640 10 monclient: init 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.875+0000 7fc906a63640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.875+0000 7fc906a63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.875+0000 7fc906a63640 10 monclient(hunting): picked mon.noname-b con 0x7fc9000a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.875+0000 7fc906a63640 10 monclient(hunting): picked mon.noname-c con 0x7fc90013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.875+0000 7fc906a63640 10 monclient(hunting): picked mon.noname-a con 0x7fc9001430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.875+0000 7fc906a63640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900137ab0 con 0x7fc9001430c0 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.875+0000 7fc906a63640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900138bd0 con 0x7fc9000a98f0 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.875+0000 7fc906a63640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900138e40 con 0x7fc90013f710 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.875+0000 7fc906a63640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:21.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.875+0000 7fc906a63640 10 monclient(hunting): authenticate will time out at 6190.056269s 2024-03-25T19:32:21.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.877+0000 7fc8ff7fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fc90013f710 0x7fc9001df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:58778/0 (socket says 172.21.15.52:58778) 2024-03-25T19:32:21.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.877+0000 7fc8ff7fe640 1 -- 172.21.15.52:0/1507110487 learned_addr learned my addr 172.21.15.52:0/1507110487 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:21.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.878+0000 7fc8fcff9640 10 client.?.objecter ms_handle_connect 0x7fc90013f710 2024-03-25T19:32:21.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.879+0000 7fc8fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:21.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.888+0000 7fc8fcff9640 10 client.?.objecter ms_handle_connect 0x7fc9001430c0 2024-03-25T19:32:21.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.889+0000 7fc8fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:21.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.890+0000 7fc8fcff9640 10 client.?.objecter ms_handle_connect 0x7fc9000a98f0 2024-03-25T19:32:21.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.890+0000 7fc8fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:21.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.894+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423390299 0 0) 0x7fc900138e40 con 0x7fc90013f710 2024-03-25T19:32:21.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.894+0000 7fc8fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.894+0000 7fc8fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.894+0000 7fc8fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.894+0000 7fc8fcff9640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:32:21.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.894+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec004eb0 con 0x7fc90013f710 2024-03-25T19:32:21.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.905+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928033016 0 0) 0x7fc900137ab0 con 0x7fc9001430c0 2024-03-25T19:32:21.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.905+0000 7fc8fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.905+0000 7fc8fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.905+0000 7fc8fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.905+0000 7fc8fcff9640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:32:21.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.905+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc900138e40 con 0x7fc9001430c0 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.918+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3492332789 0 0) 0x7fc8ec004eb0 con 0x7fc90013f710 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.918+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc900137ab0 con 0x7fc90013f710 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.918+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc8f40019b0 con 0x7fc90013f710 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.918+0000 7fc8fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.918+0000 7fc8fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.918+0000 7fc8fcff9640 10 monclient(hunting): dump: 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:21.919 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:21.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.918+0000 7fc8fcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:21.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.919+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880104218 0 0) 0x7fc900138bd0 con 0x7fc9000a98f0 2024-03-25T19:32:21.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.919+0000 7fc8fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.919+0000 7fc8fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.919+0000 7fc8fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.919+0000 7fc8fcff9640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:32:21.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.919+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8ec004eb0 con 0x7fc9000a98f0 2024-03-25T19:32:21.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.934+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2763522807 0 0) 0x7fc8ec004eb0 con 0x7fc9000a98f0 2024-03-25T19:32:21.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.934+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc900138bd0 con 0x7fc9000a98f0 2024-03-25T19:32:21.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.934+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc8f0001b00 con 0x7fc9000a98f0 2024-03-25T19:32:21.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.934+0000 7fc8fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.934+0000 7fc8fcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:21.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.934+0000 7fc8fcff9640 10 monclient(hunting): dump: 2024-03-25T19:32:21.935 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:21.935 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:21.935 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:21.935 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:21.935 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.935 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:21.935 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:21.935 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:21.935 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:21.935 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:21.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3684491726 0 0) 0x7fc900138bd0 con 0x7fc9000a98f0 2024-03-25T19:32:21.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 >> v1:172.21.15.52:6790/0 conn(0x7fc90013f710 legacy=0x7fc9001df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 >> v1:172.21.15.52:6789/0 conn(0x7fc9001430c0 legacy=0x7fc9001e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc8fcff9640 10 monclient: _finish_hunting 0 2024-03-25T19:32:21.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc8fcff9640 1 monclient: found mon.b 2024-03-25T19:32:21.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc8fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc8fcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900139120 con 0x7fc9000a98f0 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc8fcff9640 10 monclient: _finish_auth 0 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc8fcff9640 10 monclient: _check_auth_tickets 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc8fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc906a63640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc906a63640 10 monclient: _renew_subs 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc906a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc906a63640 1 -- 172.21.15.52:0/1507110487 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc900139400 con 0x7fc9000a98f0 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc906a63640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc906a63640 10 monclient: _renew_subs 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc906a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc906a63640 1 -- 172.21.15.52:0/1507110487 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9000b7880 con 0x7fc9000a98f0 2024-03-25T19:32:21.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.944+0000 7fc906a63640 1 librados: init done 2024-03-25T19:32:21.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.945+0000 7fc906a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:21.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.945+0000 7fc906a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:21.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.945+0000 7fc906a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:21.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:21.945+0000 7fc906a63640 1 -- 172.21.15.52:0/1507110487 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900137810 con 0x7fc9000a98f0 2024-03-25T19:32:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.227+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8f00038a0 con 0x7fc9000a98f0 2024-03-25T19:32:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.227+0000 7fc8fcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.227+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc8f0003bb0 con 0x7fc9000a98f0 2024-03-25T19:32:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.227+0000 7fc8fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.227+0000 7fc8fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:22.228 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.227+0000 7fc8fcff9640 10 monclient: dump: 2024-03-25T19:32:22.228 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:22.229 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:22.229 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:22.229 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:22.229 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:22.229 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:22.229 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:22.229 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:22.229 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:22.229 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:22.229 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.227+0000 7fc8fe7fc640 4 set_mon_vals no callback set 2024-03-25T19:32:22.230 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.228+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fc8f0019b80 con 0x7fc9000a98f0 2024-03-25T19:32:22.230 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.229+0000 7fc8ff7fe640 10 monclient: get_auth_request con 0x7fc8ec06df50 auth_method 0 2024-03-25T19:32:22.230 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.229+0000 7fc8fcff9640 10 client.4450.objecter ms_handle_connect 0x7fc8ec06df50 2024-03-25T19:32:22.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.233+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fc8f0053670 con 0x7fc9000a98f0 2024-03-25T19:32:22.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.233+0000 7fc8fcff9640 10 client.4450.objecter ms_dispatch 0x7fc900000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:22.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.233+0000 7fc8fcff9640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:22.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.233+0000 7fc8fcff9640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:22.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.233+0000 7fc8fcff9640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-25T19:32:22.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.234+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8f00b08b0 con 0x7fc9000a98f0 2024-03-25T19:32:22.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.234+0000 7fc8fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:22.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.234+0000 7fc8fcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:22.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.487+0000 7fc906a63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:22.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.487+0000 7fc906a63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:22.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.487+0000 7fc906a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:22.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.487+0000 7fc906a63640 1 -- 172.21.15.52:0/1507110487 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9000627b0 con 0x7fc9000a98f0 2024-03-25T19:32:22.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.517+0000 7fc8fcff9640 1 -- 172.21.15.52:0/1507110487 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fc8f004f180 con 0x7fc9000a98f0 2024-03-25T19:32:22.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.517+0000 7fc8fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:22.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.517+0000 7fc8fcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:22.518 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:32:22.518 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.519+0000 7fc906a63640 10 librados: watch_flush enter 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.519+0000 7fc906a63640 10 librados: watch_flush exit 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.519+0000 7fc906a63640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.519+0000 7fc906a63640 10 client.4450.objecter successfully canceled tick 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.519+0000 7fc906a63640 1 -- 172.21.15.52:0/1507110487 >> v1:172.21.15.52:6800/3967427147 conn(0x7fc8ec06df50 legacy=0x7fc8ec070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.519+0000 7fc906a63640 10 monclient: shutdown 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.519+0000 7fc906a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.519+0000 7fc906a63640 1 -- 172.21.15.52:0/1507110487 >> v1:172.21.15.179:6789/0 conn(0x7fc9000a98f0 legacy=0x7fc900137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.520+0000 7fc906a63640 1 -- 172.21.15.52:0/1507110487 shutdown_connections 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.520+0000 7fc906a63640 1 -- 172.21.15.52:0/1507110487 >> 172.21.15.52:0/1507110487 conn(0x7fc9000b7ea0 msgr2=0x7fc900074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.520+0000 7fc906a63640 1 -- 172.21.15.52:0/1507110487 shutdown_connections 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.520+0000 7fc906a63640 1 -- 172.21.15.52:0/1507110487 wait complete. 2024-03-25T19:32:22.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:22.520+0000 7fc906a63640 1 librados: shutdown 2024-03-25T19:32:22.533 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:26.923 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:05:20.522 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:26.929 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:05:20.528 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:26.958 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:05:20.557 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:26.975 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:05:20.517 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:26.979 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:05:20.521 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:26.982 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:05:20.581 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:26.988 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:05:20.587 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:27.019 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:05:20.618 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:27.023 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:05:20.565 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:29.339 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:29.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.435+0000 7f701813a640 10 monclient: get_monmap_and_config 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.435+0000 7f701813a640 10 monclient: build_initial_monmap 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.435+0000 7f701813a640 10 monclient: monmap: 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:32:29.437015+0000 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:32:29.437015+0000 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:32:29.436 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:29.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.436+0000 7f701813a640 1 Processor -- start 2024-03-25T19:32:29.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.436+0000 7f701813a640 1 -- start start 2024-03-25T19:32:29.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.436+0000 7f701813a640 10 monclient: init 2024-03-25T19:32:29.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f701813a640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:29.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f701813a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:29.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f701813a640 10 monclient(hunting): picked mon.noname-b con 0x7f70100b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:29.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f701813a640 10 monclient(hunting): picked mon.noname-a con 0x7f70100b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:29.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f701813a640 10 monclient(hunting): picked mon.noname-c con 0x7f701014be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:29.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f701813a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70100af940 con 0x7f70100b1730 2024-03-25T19:32:29.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f701813a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70100b0a70 con 0x7f701014be90 2024-03-25T19:32:29.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f701813a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70100b0d50 con 0x7f70100b4d60 2024-03-25T19:32:29.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f701813a640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:29.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f701813a640 10 monclient(hunting): authenticate will time out at 6197.618068s 2024-03-25T19:32:29.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f7015ed6640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f70100b4d60 0x7f701013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:49764/0 (socket says 172.21.15.52:49764) 2024-03-25T19:32:29.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.437+0000 7f7015ed6640 1 -- 172.21.15.52:0/460524102 learned_addr learned my addr 172.21.15.52:0/460524102 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:29.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.449+0000 7f7014ed4640 1 -- 172.21.15.52:0/460524102 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966410752 0 0) 0x7f70100b0d50 con 0x7f70100b4d60 2024-03-25T19:32:29.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.449+0000 7f7014ed4640 10 monclient(hunting): _negotiate 2024-03-25T19:32:29.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.449+0000 7f7014ed4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:29.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.449+0000 7f7014ed4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:29.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.449+0000 7f7014ed4640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:32:29.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.449+0000 7f7014ed4640 1 -- 172.21.15.52:0/460524102 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ffc003520 con 0x7f70100b4d60 2024-03-25T19:32:29.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 1 -- 172.21.15.52:0/460524102 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3196106729 0 0) 0x7f6ffc003520 con 0x7f70100b4d60 2024-03-25T19:32:29.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 1 -- 172.21.15.52:0/460524102 >> v1:172.21.15.179:6789/0 conn(0x7f701014be90 legacy=0x7f701014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 1 -- 172.21.15.52:0/460524102 >> v1:172.21.15.52:6789/0 conn(0x7f70100b1730 legacy=0x7f70100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 10 monclient: _finish_hunting 0 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 1 monclient: found mon.noname-b 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 1 -- 172.21.15.52:0/460524102 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70100b1030 con 0x7f70100b4d60 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 10 monclient: _finish_auth 0 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 10 monclient: _check_auth_tickets 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 1 -- 172.21.15.52:0/460524102 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f700c001690 con 0x7f70100b4d60 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f7014ed4640 10 monclient: dump: 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:29.460 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:29.461 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.461 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.461 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.461 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:29.461 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:29.461 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:29.461 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:29.461 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:29.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f701813a640 5 monclient: authenticate success, global_id 4442 2024-03-25T19:32:29.461 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.459+0000 7f701813a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:29.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.464+0000 7f7014ed4640 1 -- 172.21.15.52:0/460524102 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f700c0017f0 con 0x7f70100b4d60 2024-03-25T19:32:29.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.464+0000 7f7014ed4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.464+0000 7f7014ed4640 1 -- 172.21.15.52:0/460524102 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f700c001c00 con 0x7f70100b4d60 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.464+0000 7f701813a640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.464+0000 7f701813a640 4 set_mon_vals no callback set 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.464+0000 7f7014ed4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.464+0000 7f7014ed4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.464+0000 7f7014ed4640 10 monclient: dump: 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:29.465 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:29.466 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.465+0000 7f701813a640 10 monclient: shutdown 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.465+0000 7f701813a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.465+0000 7f701813a640 1 -- 172.21.15.52:0/460524102 >> v1:172.21.15.52:6790/0 conn(0x7f70100b4d60 legacy=0x7f701013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.465+0000 7f701813a640 1 -- 172.21.15.52:0/460524102 shutdown_connections 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.465+0000 7f701813a640 1 -- 172.21.15.52:0/460524102 >> 172.21.15.52:0/460524102 conn(0x7f70100617a0 msgr2=0x7f7010061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.465+0000 7f701813a640 1 -- 172.21.15.52:0/460524102 shutdown_connections 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.465+0000 7f701813a640 1 -- 172.21.15.52:0/460524102 wait complete. 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.466+0000 7f701813a640 10 monclient: build_initial_monmap 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.466+0000 7f701813a640 10 monclient: monmap: 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:29.467 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.466+0000 7f701813a640 1 librados: starting msgr at 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.466+0000 7f701813a640 1 librados: starting objecter 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.466+0000 7f701813a640 1 Processor -- start 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 1 -- start start 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 1 librados: setting wanted keys 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 1 librados: calling monclient init 2024-03-25T19:32:29.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 10 monclient: init 2024-03-25T19:32:29.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:29.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:29.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 10 monclient(hunting): picked mon.noname-b con 0x7f70100b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:29.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 10 monclient(hunting): picked mon.noname-c con 0x7f70100b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:29.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 10 monclient(hunting): picked mon.noname-a con 0x7f701014be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:29.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70101dbff0 con 0x7f701014be90 2024-03-25T19:32:29.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70101327f0 con 0x7f70100b1730 2024-03-25T19:32:29.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7010132f60 con 0x7f70100b4d60 2024-03-25T19:32:29.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:29.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.467+0000 7f701813a640 10 monclient(hunting): authenticate will time out at 6197.648704s 2024-03-25T19:32:29.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.485+0000 7f70156d5640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f70100b4d60 0x7f7010137130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:49784/0 (socket says 172.21.15.52:49784) 2024-03-25T19:32:29.485 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.485+0000 7f70156d5640 1 -- 172.21.15.52:0/2011454340 learned_addr learned my addr 172.21.15.52:0/2011454340 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:29.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.486+0000 7f70067fc640 10 client.?.objecter ms_handle_connect 0x7f70100b4d60 2024-03-25T19:32:29.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.486+0000 7f70067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:29.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.491+0000 7f70067fc640 10 client.?.objecter ms_handle_connect 0x7f70100b1730 2024-03-25T19:32:29.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.491+0000 7f70067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:29.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.492+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355572359 0 0) 0x7f7010132f60 con 0x7f70100b4d60 2024-03-25T19:32:29.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.492+0000 7f70067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:29.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.492+0000 7f70067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:29.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.492+0000 7f70067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:29.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.492+0000 7f70067fc640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:32:29.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.492+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff4004eb0 con 0x7f70100b4d60 2024-03-25T19:32:29.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.499+0000 7f70067fc640 10 client.?.objecter ms_handle_connect 0x7f701014be90 2024-03-25T19:32:29.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.499+0000 7f70067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:29.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.501+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415899175 0 0) 0x7f70101327f0 con 0x7f70100b1730 2024-03-25T19:32:29.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.501+0000 7f70067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:29.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.501+0000 7f70067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:29.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.501+0000 7f70067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:29.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.501+0000 7f70067fc640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:32:29.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.501+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7010132f60 con 0x7f70100b1730 2024-03-25T19:32:29.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.508+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283101211 0 0) 0x7f70101dbff0 con 0x7f701014be90 2024-03-25T19:32:29.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.508+0000 7f70067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:29.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.508+0000 7f70067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:29.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.508+0000 7f70067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:29.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.508+0000 7f70067fc640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:32:29.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.508+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70101327f0 con 0x7f701014be90 2024-03-25T19:32:29.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.517+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 650355746 0 0) 0x7f7010132f60 con 0x7f70100b1730 2024-03-25T19:32:29.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.517+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70101dbff0 con 0x7f70100b1730 2024-03-25T19:32:29.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.517+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f700c001b40 con 0x7f70100b1730 2024-03-25T19:32:29.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.517+0000 7f70067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.517+0000 7f70067fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.517+0000 7f70067fc640 10 monclient(hunting): dump: 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:29.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.517+0000 7f70067fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:29.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.518+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1724954544 0 0) 0x7f70101327f0 con 0x7f701014be90 2024-03-25T19:32:29.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.518+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7010132f60 con 0x7f701014be90 2024-03-25T19:32:29.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.518+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7008001a90 con 0x7f701014be90 2024-03-25T19:32:29.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.518+0000 7f70067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.518+0000 7f70067fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:29.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.518+0000 7f70067fc640 10 monclient(hunting): dump: 2024-03-25T19:32:29.519 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:29.519 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:29.519 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.520 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.520 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.520 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:29.520 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:29.520 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:29.520 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:29.520 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.524+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3551694856 0 0) 0x7f6ff4004eb0 con 0x7f70100b4d60 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.524+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70101327f0 con 0x7f70100b4d60 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.524+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f70000019d0 con 0x7f70100b4d60 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.524+0000 7f70067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.524+0000 7f70067fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.524+0000 7f70067fc640 10 monclient(hunting): dump: 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.525 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:29.526 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:29.526 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:29.526 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:29.526 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:29.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1331646688 0 0) 0x7f70101327f0 con 0x7f70100b4d60 2024-03-25T19:32:29.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 >> v1:172.21.15.179:6789/0 conn(0x7f70100b1730 legacy=0x7f701012da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 >> v1:172.21.15.52:6789/0 conn(0x7f701014be90 legacy=0x7f70101e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f70067fc640 10 monclient: _finish_hunting 0 2024-03-25T19:32:29.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f70067fc640 1 monclient: found mon.c 2024-03-25T19:32:29.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f70067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:29.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f70067fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:32:29.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701012f5b0 con 0x7f70100b4d60 2024-03-25T19:32:29.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f70067fc640 10 monclient: _finish_auth 0 2024-03-25T19:32:29.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f70067fc640 10 monclient: _check_auth_tickets 2024-03-25T19:32:29.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f70067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:29.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f701813a640 5 monclient: authenticate success, global_id 4445 2024-03-25T19:32:29.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f701813a640 10 monclient: _renew_subs 2024-03-25T19:32:29.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f701813a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:32:29.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f701813a640 1 -- 172.21.15.52:0/2011454340 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701012f900 con 0x7f70100b4d60 2024-03-25T19:32:29.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f701813a640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:29.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f701813a640 10 monclient: _renew_subs 2024-03-25T19:32:29.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f701813a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:32:29.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f701813a640 1 -- 172.21.15.52:0/2011454340 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70101db090 con 0x7f70100b4d60 2024-03-25T19:32:29.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.535+0000 7f701813a640 1 librados: init done 2024-03-25T19:32:29.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.536+0000 7f701813a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:29.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.536+0000 7f701813a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:29.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.536+0000 7f701813a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:32:29.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.536+0000 7f701813a640 1 -- 172.21.15.52:0/2011454340 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70100627b0 con 0x7f70100b4d60 2024-03-25T19:32:29.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.849+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7000003610 con 0x7f70100b4d60 2024-03-25T19:32:29.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.849+0000 7f70067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:29.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.849+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f70000039e0 con 0x7f70100b4d60 2024-03-25T19:32:29.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.849+0000 7f70067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.849+0000 7f70067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:29.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.849+0000 7f70067fc640 10 monclient: dump: 2024-03-25T19:32:29.851 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:29.851 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:29.851 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.851 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:29.851 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.851 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:29.851 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:29.851 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:29.851 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:29.851 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:29.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.849+0000 7f7007fff640 4 set_mon_vals no callback set 2024-03-25T19:32:29.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.850+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f7000003c00 con 0x7f70100b4d60 2024-03-25T19:32:29.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.851+0000 7f7015ed6640 10 monclient: get_auth_request con 0x7f6ff406e160 auth_method 0 2024-03-25T19:32:29.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.851+0000 7f70067fc640 10 client.4445.objecter ms_handle_connect 0x7f6ff406e160 2024-03-25T19:32:29.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.856+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f7000054800 con 0x7f70100b4d60 2024-03-25T19:32:29.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.856+0000 7f70067fc640 10 client.4445.objecter ms_dispatch 0x7f7010000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:29.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.856+0000 7f70067fc640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:29.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.856+0000 7f70067fc640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:29.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.856+0000 7f70067fc640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-25T19:32:29.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.856+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7000050310 con 0x7f70100b4d60 2024-03-25T19:32:29.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.856+0000 7f70067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:29.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:29.856+0000 7f70067fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:30.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.126+0000 7f701813a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:30.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.126+0000 7f701813a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:30.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.126+0000 7f701813a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:32:30.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.126+0000 7f701813a640 1 -- 172.21.15.52:0/2011454340 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f701012f740 con 0x7f70100b4d60 2024-03-25T19:32:30.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.156+0000 7f70067fc640 1 -- 172.21.15.52:0/2011454340 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f701012f740 con 0x7f70100b4d60 2024-03-25T19:32:30.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.156+0000 7f70067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:30.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.156+0000 7f70067fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:30.157 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:32:30.157 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.158+0000 7f701813a640 10 librados: watch_flush enter 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 10 librados: watch_flush exit 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 10 client.4445.objecter successfully canceled tick 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 1 -- 172.21.15.52:0/2011454340 >> v1:172.21.15.52:6800/3967427147 conn(0x7f6ff406e160 legacy=0x7f6ff4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 10 monclient: shutdown 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 1 -- 172.21.15.52:0/2011454340 >> v1:172.21.15.52:6790/0 conn(0x7f70100b4d60 legacy=0x7f7010137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 1 -- 172.21.15.52:0/2011454340 shutdown_connections 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 1 -- 172.21.15.52:0/2011454340 >> 172.21.15.52:0/2011454340 conn(0x7f70100617a0 msgr2=0x7f70100b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 1 -- 172.21.15.52:0/2011454340 shutdown_connections 2024-03-25T19:32:30.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 1 -- 172.21.15.52:0/2011454340 wait complete. 2024-03-25T19:32:30.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:30.159+0000 7f701813a640 1 librados: shutdown 2024-03-25T19:32:30.172 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:36.979 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:37.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.095+0000 7f28ee1bc640 10 monclient: get_monmap_and_config 2024-03-25T19:32:37.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.095+0000 7f28ee1bc640 10 monclient: build_initial_monmap 2024-03-25T19:32:37.095 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.095+0000 7f28ee1bc640 10 monclient: monmap: 2024-03-25T19:32:37.095 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:37.095 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:37.095 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:32:37.096340+0000 2024-03-25T19:32:37.096 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:32:37.096340+0000 2024-03-25T19:32:37.096 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:37.096 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:37.096 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:37.096 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:32:37.096 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:32:37.096 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:37.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.096+0000 7f28ee1bc640 1 Processor -- start 2024-03-25T19:32:37.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.096+0000 7f28ee1bc640 1 -- start start 2024-03-25T19:32:37.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.096+0000 7f28ee1bc640 10 monclient: init 2024-03-25T19:32:37.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.097+0000 7f28ee1bc640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:37.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.097+0000 7f28ee1bc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:37.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.097+0000 7f28ee1bc640 10 monclient(hunting): picked mon.noname-b con 0x7f28e80a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:37.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.097+0000 7f28ee1bc640 10 monclient(hunting): picked mon.noname-a con 0x7f28e813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:37.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.097+0000 7f28ee1bc640 10 monclient(hunting): picked mon.noname-c con 0x7f28e81430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:37.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.097+0000 7f28ee1bc640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e8142520 con 0x7f28e813f710 2024-03-25T19:32:37.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.097+0000 7f28ee1bc640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e80b28a0 con 0x7f28e81430c0 2024-03-25T19:32:37.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.097+0000 7f28ee1bc640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e80b2a30 con 0x7f28e80a98f0 2024-03-25T19:32:37.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.097+0000 7f28ee1bc640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:37.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.097+0000 7f28ee1bc640 10 monclient(hunting): authenticate will time out at 6205.277494s 2024-03-25T19:32:37.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.103+0000 7f28e7fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f28e81430c0 0x7f28e814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:57036/0 (socket says 172.21.15.52:57036) 2024-03-25T19:32:37.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.103+0000 7f28e7fff640 1 -- 172.21.15.52:0/3952851007 learned_addr learned my addr 172.21.15.52:0/3952851007 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:37.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.115+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944595332 0 0) 0x7f28e80b2a30 con 0x7f28e80a98f0 2024-03-25T19:32:37.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.115+0000 7f28e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:37.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.115+0000 7f28e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:37.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.115+0000 7f28e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:37.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.115+0000 7f28e67fc640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:32:37.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.115+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d0003520 con 0x7f28e80a98f0 2024-03-25T19:32:37.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.117+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897299329 0 0) 0x7f28e80b28a0 con 0x7f28e81430c0 2024-03-25T19:32:37.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.117+0000 7f28e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:37.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.117+0000 7f28e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:37.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.117+0000 7f28e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:37.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.117+0000 7f28e67fc640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:32:37.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.117+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e80b2a30 con 0x7f28e81430c0 2024-03-25T19:32:37.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.127+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979861244 0 0) 0x7f28e8142520 con 0x7f28e813f710 2024-03-25T19:32:37.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.127+0000 7f28e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:37.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.127+0000 7f28e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:37.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.127+0000 7f28e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:37.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.127+0000 7f28e67fc640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:32:37.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.127+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e80b28a0 con 0x7f28e813f710 2024-03-25T19:32:37.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3514062241 0 0) 0x7f28e80b28a0 con 0x7f28e813f710 2024-03-25T19:32:37.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 >> v1:172.21.15.52:6790/0 conn(0x7f28e80a98f0 legacy=0x7f28e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 >> v1:172.21.15.179:6789/0 conn(0x7f28e81430c0 legacy=0x7f28e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:32:37.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 1 monclient: found mon.noname-a 2024-03-25T19:32:37.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e80b44c0 con 0x7f28e813f710 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 10 monclient: _finish_auth 0 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f28dc001570 con 0x7f28e813f710 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28e67fc640 10 monclient: dump: 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:37.137 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:37.138 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:37.138 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:37.138 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:37.138 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:37.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28ee1bc640 5 monclient: authenticate success, global_id 4452 2024-03-25T19:32:37.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.136+0000 7f28ee1bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:37.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.149+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28dc0016d0 con 0x7f28e813f710 2024-03-25T19:32:37.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.149+0000 7f28e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:37.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.149+0000 7f28e67fc640 1 -- 172.21.15.52:0/3952851007 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f28dc001b20 con 0x7f28e813f710 2024-03-25T19:32:37.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.149+0000 7f28ee1bc640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:37.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.149+0000 7f28ee1bc640 4 set_mon_vals no callback set 2024-03-25T19:32:37.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.149+0000 7f28e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:37.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.149+0000 7f28e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:37.150 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.149+0000 7f28e67fc640 10 monclient: dump: 2024-03-25T19:32:37.150 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:37.150 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:37.150 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:37.150 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:37.150 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:37.150 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:37.150 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:37.150 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:37.150 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:37.150 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:37.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.150+0000 7f28ee1bc640 10 monclient: shutdown 2024-03-25T19:32:37.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.150+0000 7f28ee1bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:37.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.150+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3952851007 >> v1:172.21.15.52:6789/0 conn(0x7f28e813f710 legacy=0x7f28e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.150+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3952851007 shutdown_connections 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.150+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3952851007 >> 172.21.15.52:0/3952851007 conn(0x7f28e80b7ea0 msgr2=0x7f28e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.150+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3952851007 shutdown_connections 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.150+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3952851007 wait complete. 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.151+0000 7f28ee1bc640 10 monclient: build_initial_monmap 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.151+0000 7f28ee1bc640 10 monclient: monmap: 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:37.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.152+0000 7f28ee1bc640 1 librados: starting msgr at 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.152+0000 7f28ee1bc640 1 librados: starting objecter 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.152+0000 7f28ee1bc640 1 Processor -- start 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.152+0000 7f28ee1bc640 1 -- start start 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.152+0000 7f28ee1bc640 1 librados: setting wanted keys 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.152+0000 7f28ee1bc640 1 librados: calling monclient init 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.152+0000 7f28ee1bc640 10 monclient: init 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.153+0000 7f28ee1bc640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.153+0000 7f28ee1bc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.153+0000 7f28ee1bc640 10 monclient(hunting): picked mon.noname-c con 0x7f28e80a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.153+0000 7f28ee1bc640 10 monclient(hunting): picked mon.noname-a con 0x7f28e813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.153+0000 7f28ee1bc640 10 monclient(hunting): picked mon.noname-b con 0x7f28e81430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.153+0000 7f28ee1bc640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e81d8340 con 0x7f28e813f710 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.153+0000 7f28ee1bc640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e81d7710 con 0x7f28e81430c0 2024-03-25T19:32:37.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.153+0000 7f28ee1bc640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e8138e90 con 0x7f28e80a98f0 2024-03-25T19:32:37.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.153+0000 7f28ee1bc640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:37.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.153+0000 7f28ee1bc640 10 monclient(hunting): authenticate will time out at 6205.333074s 2024-03-25T19:32:37.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.154+0000 7f28e7fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f28e81430c0 0x7f28e8133ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:57074/0 (socket says 172.21.15.52:57074) 2024-03-25T19:32:37.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.154+0000 7f28e7fff640 1 -- 172.21.15.52:0/3577563191 learned_addr learned my addr 172.21.15.52:0/3577563191 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:37.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.154+0000 7f28cffff640 10 client.?.objecter ms_handle_connect 0x7f28e81430c0 2024-03-25T19:32:37.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.154+0000 7f28cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:37.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.163+0000 7f28cffff640 10 client.?.objecter ms_handle_connect 0x7f28e80a98f0 2024-03-25T19:32:37.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.163+0000 7f28cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:37.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.174+0000 7f28cffff640 10 client.?.objecter ms_handle_connect 0x7f28e813f710 2024-03-25T19:32:37.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.174+0000 7f28cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:37.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.180+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090657254 0 0) 0x7f28e81d8340 con 0x7f28e813f710 2024-03-25T19:32:37.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.180+0000 7f28cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:37.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.180+0000 7f28cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:37.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.180+0000 7f28cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:37.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.180+0000 7f28cffff640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:32:37.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.180+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d4004eb0 con 0x7f28e813f710 2024-03-25T19:32:37.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.247+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963468837 0 0) 0x7f28e8138e90 con 0x7f28e80a98f0 2024-03-25T19:32:37.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.247+0000 7f28cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:37.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.247+0000 7f28cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:37.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.247+0000 7f28cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:37.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.247+0000 7f28cffff640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:32:37.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.247+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e81d8340 con 0x7f28e80a98f0 2024-03-25T19:32:37.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.250+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782543882 0 0) 0x7f28e81d7710 con 0x7f28e81430c0 2024-03-25T19:32:37.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.251+0000 7f28cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:37.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.251+0000 7f28cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:37.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.251+0000 7f28cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:37.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.251+0000 7f28cffff640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:32:37.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.251+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e8138e90 con 0x7f28e81430c0 2024-03-25T19:32:37.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.255+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1251592706 0 0) 0x7f28d4004eb0 con 0x7f28e813f710 2024-03-25T19:32:37.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.255+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28e81d7710 con 0x7f28e813f710 2024-03-25T19:32:37.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.255+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f28dc001a60 con 0x7f28e813f710 2024-03-25T19:32:37.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.255+0000 7f28cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:37.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.255+0000 7f28cffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:37.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.255+0000 7f28cffff640 10 monclient(hunting): dump: 2024-03-25T19:32:37.259 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:37.259 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:37.259 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:37.259 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:37.259 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.255+0000 7f28cffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.258+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4189761613 0 0) 0x7f28e8138e90 con 0x7f28e81430c0 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.258+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28d4004eb0 con 0x7f28e81430c0 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.258+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f28e0001ad0 con 0x7f28e81430c0 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.258+0000 7f28cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.258+0000 7f28cffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.258+0000 7f28cffff640 10 monclient(hunting): dump: 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:37.260 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:37.261 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:37.261 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:37.261 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:37.261 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:37.261 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:37.261 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:37.261 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:37.261 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:37.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3828969574 0 0) 0x7f28e81d7710 con 0x7f28e813f710 2024-03-25T19:32:37.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 >> v1:172.21.15.52:6790/0 conn(0x7f28e80a98f0 legacy=0x7f28e81d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 >> v1:172.21.15.179:6789/0 conn(0x7f28e81430c0 legacy=0x7f28e8133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28cffff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28cffff640 1 monclient: found mon.a 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28cffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e8139020 con 0x7f28e813f710 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28cffff640 10 monclient: _finish_auth 0 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28cffff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28ee1bc640 5 monclient: authenticate success, global_id 4455 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28ee1bc640 10 monclient: _renew_subs 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28ee1bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.265+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3577563191 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e8137f30 con 0x7f28e813f710 2024-03-25T19:32:37.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.266+0000 7f28ee1bc640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:37.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.266+0000 7f28ee1bc640 10 monclient: _renew_subs 2024-03-25T19:32:37.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.266+0000 7f28ee1bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:37.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.266+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3577563191 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e81348e0 con 0x7f28e813f710 2024-03-25T19:32:37.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.266+0000 7f28ee1bc640 1 librados: init done 2024-03-25T19:32:37.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.267+0000 7f28ee1bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:37.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.267+0000 7f28ee1bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:37.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.267+0000 7f28ee1bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:37.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.267+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3577563191 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e81d78a0 con 0x7f28e813f710 2024-03-25T19:32:37.607 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.607+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28dc0038a0 con 0x7f28e813f710 2024-03-25T19:32:37.607 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.607+0000 7f28cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:37.607 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.607+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f28dc003bb0 con 0x7f28e813f710 2024-03-25T19:32:37.607 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.607+0000 7f28cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:37.607 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.607+0000 7f28cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:37.607 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.607+0000 7f28cffff640 10 monclient: dump: 2024-03-25T19:32:37.607 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:37.607 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:37.607 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:37.608 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:37.608 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:37.608 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:37.608 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:37.608 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:37.608 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:37.608 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:37.608 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.607+0000 7f28e5ffb640 4 set_mon_vals no callback set 2024-03-25T19:32:37.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.608+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f28dc019b80 con 0x7f28e813f710 2024-03-25T19:32:37.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.608+0000 7f28e77fe640 10 monclient: get_auth_request con 0x7f28d406df90 auth_method 0 2024-03-25T19:32:37.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.609+0000 7f28cffff640 10 client.4455.objecter ms_handle_connect 0x7f28d406df90 2024-03-25T19:32:37.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.614+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f28dc0536a0 con 0x7f28e813f710 2024-03-25T19:32:37.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.614+0000 7f28cffff640 10 client.4455.objecter ms_dispatch 0x7f28e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:37.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.614+0000 7f28cffff640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:37.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.614+0000 7f28cffff640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:37.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.614+0000 7f28cffff640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-25T19:32:37.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.614+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f28dc04f1b0 con 0x7f28e813f710 2024-03-25T19:32:37.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.614+0000 7f28cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:37.614 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.614+0000 7f28cffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:37.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.867+0000 7f28ee1bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:37.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.867+0000 7f28ee1bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:37.869 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.867+0000 7f28ee1bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:37.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.867+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3577563191 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28e814f2d0 con 0x7f28e813f710 2024-03-25T19:32:37.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.893+0000 7f28cffff640 1 -- 172.21.15.52:0/3577563191 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f28dc0510e0 con 0x7f28e813f710 2024-03-25T19:32:37.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.893+0000 7f28cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:37.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.893+0000 7f28cffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:37.893 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:32:37.893 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.895+0000 7f28ee1bc640 10 librados: watch_flush enter 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 10 librados: watch_flush exit 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 10 client.4455.objecter successfully canceled tick 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3577563191 >> v1:172.21.15.52:6800/3967427147 conn(0x7f28d406df90 legacy=0x7f28d4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 10 monclient: shutdown 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3577563191 >> v1:172.21.15.52:6789/0 conn(0x7f28e813f710 legacy=0x7f28e8138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3577563191 shutdown_connections 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3577563191 >> 172.21.15.52:0/3577563191 conn(0x7f28e80b7ea0 msgr2=0x7f28e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3577563191 shutdown_connections 2024-03-25T19:32:37.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 1 -- 172.21.15.52:0/3577563191 wait complete. 2024-03-25T19:32:37.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:37.896+0000 7f28ee1bc640 1 librados: shutdown 2024-03-25T19:32:37.909 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:44.716 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:44.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.816+0000 7f8d74fd6640 10 monclient: get_monmap_and_config 2024-03-25T19:32:44.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.816+0000 7f8d74fd6640 10 monclient: build_initial_monmap 2024-03-25T19:32:44.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.816+0000 7f8d74fd6640 10 monclient: monmap: 2024-03-25T19:32:44.819 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:44.819 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:44.819 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:32:44.817437+0000 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:32:44.817437+0000 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.817+0000 7f8d74fd6640 1 Processor -- start 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.817+0000 7f8d74fd6640 1 -- start start 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.817+0000 7f8d74fd6640 10 monclient: init 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d74fd6640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d74fd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d74fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f8d700b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d74fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f8d700b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:44.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d74fd6640 10 monclient(hunting): picked mon.noname-b con 0x7f8d7014be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:44.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d74fd6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700af940 con 0x7f8d700b4d60 2024-03-25T19:32:44.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d74fd6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700b0a70 con 0x7f8d700b1730 2024-03-25T19:32:44.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d74fd6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d700b0d50 con 0x7f8d7014be90 2024-03-25T19:32:44.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d74fd6640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:44.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d74fd6640 10 monclient(hunting): authenticate will time out at 6212.998424s 2024-03-25T19:32:44.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d6ed9d640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f8d7014be90 0x7f8d7014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:33308/0 (socket says 172.21.15.52:33308) 2024-03-25T19:32:44.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.818+0000 7f8d6ed9d640 1 -- 172.21.15.52:0/2652056276 learned_addr learned my addr 172.21.15.52:0/2652056276 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:44.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.842+0000 7f8d6dd9b640 1 -- 172.21.15.52:0/2652056276 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368136433 0 0) 0x7f8d700af940 con 0x7f8d700b4d60 2024-03-25T19:32:44.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.842+0000 7f8d6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:32:44.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.842+0000 7f8d6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:44.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.842+0000 7f8d6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:44.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.842+0000 7f8d6dd9b640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:32:44.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.842+0000 7f8d6dd9b640 1 -- 172.21.15.52:0/2652056276 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c003520 con 0x7f8d700b4d60 2024-03-25T19:32:44.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 1 -- 172.21.15.52:0/2652056276 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2181473063 0 0) 0x7f8d5c003520 con 0x7f8d700b4d60 2024-03-25T19:32:44.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 1 -- 172.21.15.52:0/2652056276 >> v1:172.21.15.52:6790/0 conn(0x7f8d7014be90 legacy=0x7f8d7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 1 -- 172.21.15.52:0/2652056276 >> v1:172.21.15.179:6789/0 conn(0x7f8d700b1730 legacy=0x7f8d700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 1 -- 172.21.15.52:0/2652056276 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d700b1030 con 0x7f8d700b4d60 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 1 -- 172.21.15.52:0/2652056276 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8d60001690 con 0x7f8d700b4d60 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d74fd6640 5 monclient: authenticate success, global_id 4458 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d74fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:44.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d6dd9b640 10 monclient: dump: 2024-03-25T19:32:44.855 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:44.855 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:44.855 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:44.855 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:44.855 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:44.855 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:44.855 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:44.855 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:44.855 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:44.855 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:44.855 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.853+0000 7f8d74fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:44.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.860+0000 7f8d6dd9b640 1 -- 172.21.15.52:0/2652056276 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d600017f0 con 0x7f8d700b4d60 2024-03-25T19:32:44.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.860+0000 7f8d6dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:44.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.860+0000 7f8d74fd6640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:44.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.860+0000 7f8d6dd9b640 1 -- 172.21.15.52:0/2652056276 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8d60001c00 con 0x7f8d700b4d60 2024-03-25T19:32:44.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.860+0000 7f8d6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:44.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.860+0000 7f8d74fd6640 4 set_mon_vals no callback set 2024-03-25T19:32:44.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.860+0000 7f8d6dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:44.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.860+0000 7f8d6dd9b640 10 monclient: dump: 2024-03-25T19:32:44.860 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:44.860 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:44.861 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:44.861 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:44.861 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:44.861 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:44.861 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:44.861 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:44.861 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:44.861 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:44.863 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.861+0000 7f8d74fd6640 10 monclient: shutdown 2024-03-25T19:32:44.863 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.861+0000 7f8d74fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:44.863 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.861+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2652056276 >> v1:172.21.15.52:6789/0 conn(0x7f8d700b4d60 legacy=0x7f8d7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.863 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.861+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2652056276 shutdown_connections 2024-03-25T19:32:44.863 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.861+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2652056276 >> 172.21.15.52:0/2652056276 conn(0x7f8d700617a0 msgr2=0x7f8d70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:44.863 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.861+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2652056276 shutdown_connections 2024-03-25T19:32:44.863 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.861+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2652056276 wait complete. 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.862+0000 7f8d74fd6640 10 monclient: build_initial_monmap 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.862+0000 7f8d74fd6640 10 monclient: monmap: 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.862+0000 7f8d74fd6640 1 librados: starting msgr at 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.862+0000 7f8d74fd6640 1 librados: starting objecter 2024-03-25T19:32:44.864 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 1 Processor -- start 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 1 -- start start 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 1 librados: setting wanted keys 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 1 librados: calling monclient init 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 10 monclient: init 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 10 monclient(hunting): picked mon.noname-a con 0x7f8d700b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 10 monclient(hunting): picked mon.noname-c con 0x7f8d700b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 10 monclient(hunting): picked mon.noname-b con 0x7f8d7014be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d701d8200 con 0x7f8d700b1730 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d70139250 con 0x7f8d7014be90 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d701362d0 con 0x7f8d700b4d60 2024-03-25T19:32:44.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:44.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.863+0000 7f8d74fd6640 10 monclient(hunting): authenticate will time out at 6213.043599s 2024-03-25T19:32:44.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.880+0000 7f8d6e59c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f8d700b1730 0x7f8d70137070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:38086/0 (socket says 172.21.15.52:38086) 2024-03-25T19:32:44.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.880+0000 7f8d6e59c640 1 -- 172.21.15.52:0/2947868065 learned_addr learned my addr 172.21.15.52:0/2947868065 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:44.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.882+0000 7f8d66ffd640 10 client.?.objecter ms_handle_connect 0x7f8d700b1730 2024-03-25T19:32:44.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.882+0000 7f8d66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:44.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.885+0000 7f8d66ffd640 10 client.?.objecter ms_handle_connect 0x7f8d7014be90 2024-03-25T19:32:44.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.885+0000 7f8d66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:44.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.900+0000 7f8d66ffd640 10 client.?.objecter ms_handle_connect 0x7f8d700b4d60 2024-03-25T19:32:44.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.900+0000 7f8d66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:44.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.901+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315987100 0 0) 0x7f8d70139250 con 0x7f8d7014be90 2024-03-25T19:32:44.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.901+0000 7f8d66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:44.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.901+0000 7f8d66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:44.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.901+0000 7f8d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:44.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.901+0000 7f8d66ffd640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:32:44.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.901+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d50004eb0 con 0x7f8d7014be90 2024-03-25T19:32:44.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.907+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086807126 0 0) 0x7f8d701d8200 con 0x7f8d700b1730 2024-03-25T19:32:44.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.907+0000 7f8d66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:44.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.907+0000 7f8d66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:44.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.907+0000 7f8d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:44.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.907+0000 7f8d66ffd640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:32:44.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.907+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d70139250 con 0x7f8d700b1730 2024-03-25T19:32:44.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.908+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085660574 0 0) 0x7f8d701362d0 con 0x7f8d700b4d60 2024-03-25T19:32:44.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.908+0000 7f8d66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:44.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.908+0000 7f8d66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:44.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.908+0000 7f8d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:44.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.908+0000 7f8d66ffd640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:32:44.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.908+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d701d8200 con 0x7f8d700b4d60 2024-03-25T19:32:44.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.917+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1978482822 0 0) 0x7f8d50004eb0 con 0x7f8d7014be90 2024-03-25T19:32:44.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.917+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d701362d0 con 0x7f8d7014be90 2024-03-25T19:32:44.926 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.927+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 889213005 0 0) 0x7f8d70139250 con 0x7f8d700b1730 2024-03-25T19:32:44.926 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.927+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d50004eb0 con 0x7f8d700b1730 2024-03-25T19:32:44.926 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.927+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8d60001690 con 0x7f8d700b1730 2024-03-25T19:32:44.926 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.927+0000 7f8d66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:44.926 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.927+0000 7f8d66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.927+0000 7f8d66ffd640 10 monclient(hunting): dump: 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:44.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.927+0000 7f8d66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:44.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.928+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8d68001b20 con 0x7f8d7014be90 2024-03-25T19:32:44.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.928+0000 7f8d66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:44.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.928+0000 7f8d66ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:44.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.928+0000 7f8d66ffd640 10 monclient(hunting): dump: 2024-03-25T19:32:44.928 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:44.928 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:44.928 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:44.928 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:44.929 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:44.929 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:44.929 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:44.929 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:44.929 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:44.929 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4170460174 0 0) 0x7f8d701362d0 con 0x7f8d7014be90 2024-03-25T19:32:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 >> v1:172.21.15.52:6790/0 conn(0x7f8d700b4d60 legacy=0x7f8d701d8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 >> v1:172.21.15.52:6789/0 conn(0x7f8d700b1730 legacy=0x7f8d70137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d66ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d66ffd640 1 monclient: found mon.b 2024-03-25T19:32:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d66ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7013be10 con 0x7f8d7014be90 2024-03-25T19:32:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d66ffd640 10 monclient: _finish_auth 0 2024-03-25T19:32:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d66ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d74fd6640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:32:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d74fd6640 10 monclient: _renew_subs 2024-03-25T19:32:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d74fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2947868065 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d70137780 con 0x7f8d7014be90 2024-03-25T19:32:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d74fd6640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d74fd6640 10 monclient: _renew_subs 2024-03-25T19:32:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d74fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2947868065 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d70143ae0 con 0x7f8d7014be90 2024-03-25T19:32:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.939+0000 7f8d74fd6640 1 librados: init done 2024-03-25T19:32:44.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.940+0000 7f8d74fd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:44.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.940+0000 7f8d74fd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:44.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.940+0000 7f8d74fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:44.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:44.940+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2947868065 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d701d83b0 con 0x7f8d7014be90 2024-03-25T19:32:45.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.210+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d68001ff0 con 0x7f8d7014be90 2024-03-25T19:32:45.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.210+0000 7f8d66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:45.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.210+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8d68004d00 con 0x7f8d7014be90 2024-03-25T19:32:45.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.210+0000 7f8d66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:45.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.210+0000 7f8d66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:45.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.210+0000 7f8d66ffd640 10 monclient: dump: 2024-03-25T19:32:45.210 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:45.210 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:45.211 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:45.211 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:45.211 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:45.211 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:45.211 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:45.211 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:45.211 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:45.211 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:45.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.210+0000 7f8d6d59a640 4 set_mon_vals no callback set 2024-03-25T19:32:45.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.211+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f8d6801acd0 con 0x7f8d7014be90 2024-03-25T19:32:45.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.212+0000 7f8d6e59c640 10 monclient: get_auth_request con 0x7f8d5006dfa0 auth_method 0 2024-03-25T19:32:45.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.212+0000 7f8d66ffd640 10 client.4468.objecter ms_handle_connect 0x7f8d5006dfa0 2024-03-25T19:32:45.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.215+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f8d68054980 con 0x7f8d7014be90 2024-03-25T19:32:45.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.215+0000 7f8d66ffd640 10 client.4468.objecter ms_dispatch 0x7f8d70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:45.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.215+0000 7f8d66ffd640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:45.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.215+0000 7f8d66ffd640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:45.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.215+0000 7f8d66ffd640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T19:32:45.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.216+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d680502c0 con 0x7f8d7014be90 2024-03-25T19:32:45.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.216+0000 7f8d66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:45.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.216+0000 7f8d66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:45.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.476+0000 7f8d74fd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:45.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.476+0000 7f8d74fd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:45.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.476+0000 7f8d74fd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:32:45.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.476+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2947868065 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d700627b0 con 0x7f8d7014be90 2024-03-25T19:32:45.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.495+0000 7f8d66ffd640 1 -- 172.21.15.52:0/2947868065 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f8d680523c0 con 0x7f8d7014be90 2024-03-25T19:32:45.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.495+0000 7f8d66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:45.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.495+0000 7f8d66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:45.495 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:32:45.495 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:45.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 10 librados: watch_flush enter 2024-03-25T19:32:45.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 10 librados: watch_flush exit 2024-03-25T19:32:45.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T19:32:45.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 10 client.4468.objecter successfully canceled tick 2024-03-25T19:32:45.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2947868065 >> v1:172.21.15.52:6800/3967427147 conn(0x7f8d5006dfa0 legacy=0x7f8d50070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:45.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 10 monclient: shutdown 2024-03-25T19:32:45.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:45.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2947868065 >> v1:172.21.15.179:6789/0 conn(0x7f8d7014be90 legacy=0x7f8d701e2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:45.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2947868065 shutdown_connections 2024-03-25T19:32:45.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2947868065 >> 172.21.15.52:0/2947868065 conn(0x7f8d700617a0 msgr2=0x7f8d700b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:45.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2947868065 shutdown_connections 2024-03-25T19:32:45.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 1 -- 172.21.15.52:0/2947868065 wait complete. 2024-03-25T19:32:45.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:45.498+0000 7f8d74fd6640 1 librados: shutdown 2024-03-25T19:32:45.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:51.997 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:05:45.596 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:52.001 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:05:45.600 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:52.039 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:05:45.638 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:52.057 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:05:45.656 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:52.070 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:05:45.612 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:52.074 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:05:45.616 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:52.074 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:05:45.673 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:52.093 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:05:45.635 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:52.118 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:05:45.717 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:52.317 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:52.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.434+0000 7f3e34b48640 10 monclient: get_monmap_and_config 2024-03-25T19:32:52.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.434+0000 7f3e34b48640 10 monclient: build_initial_monmap 2024-03-25T19:32:52.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.434+0000 7f3e34b48640 10 monclient: monmap: 2024-03-25T19:32:52.435 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:52.435 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:52.435 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:32:52.435303+0000 2024-03-25T19:32:52.435 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:32:52.435303+0000 2024-03-25T19:32:52.435 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:52.435 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:52.435 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:52.435 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:32:52.435 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:32:52.435 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:52.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.435+0000 7f3e34b48640 1 Processor -- start 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.435+0000 7f3e34b48640 1 -- start start 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.435+0000 7f3e34b48640 10 monclient: init 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.436+0000 7f3e34b48640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.436+0000 7f3e34b48640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.436+0000 7f3e34b48640 10 monclient(hunting): picked mon.noname-a con 0x7f3e300a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.436+0000 7f3e34b48640 10 monclient(hunting): picked mon.noname-b con 0x7f3e3013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.436+0000 7f3e34b48640 10 monclient(hunting): picked mon.noname-c con 0x7f3e301430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.436+0000 7f3e34b48640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e30142520 con 0x7f3e300a98f0 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.436+0000 7f3e34b48640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300b28a0 con 0x7f3e301430c0 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.436+0000 7f3e34b48640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e300b2a30 con 0x7f3e3013f710 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.436+0000 7f3e34b48640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:52.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.436+0000 7f3e34b48640 10 monclient(hunting): authenticate will time out at 6220.616452s 2024-03-25T19:32:52.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.444+0000 7f3e2e59c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f3e300a98f0 0x7f3e300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:42630/0 (socket says 172.21.15.52:42630) 2024-03-25T19:32:52.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.444+0000 7f3e2e59c640 1 -- 172.21.15.52:0/1765542647 learned_addr learned my addr 172.21.15.52:0/1765542647 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:52.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.456+0000 7f3e2d59a640 1 -- 172.21.15.52:0/1765542647 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500625166 0 0) 0x7f3e30142520 con 0x7f3e300a98f0 2024-03-25T19:32:52.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.456+0000 7f3e2d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:32:52.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.456+0000 7f3e2d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:52.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.456+0000 7f3e2d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:52.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.456+0000 7f3e2d59a640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:32:52.456 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.456+0000 7f3e2d59a640 1 -- 172.21.15.52:0/1765542647 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e18003520 con 0x7f3e300a98f0 2024-03-25T19:32:52.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.459+0000 7f3e2d59a640 1 -- 172.21.15.52:0/1765542647 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954663205 0 0) 0x7f3e300b2a30 con 0x7f3e3013f710 2024-03-25T19:32:52.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.459+0000 7f3e2d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:32:52.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.459+0000 7f3e2d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:52.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.459+0000 7f3e2d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:52.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.459+0000 7f3e2d59a640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:32:52.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.459+0000 7f3e2d59a640 1 -- 172.21.15.52:0/1765542647 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e30142520 con 0x7f3e3013f710 2024-03-25T19:32:52.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 1 -- 172.21.15.52:0/1765542647 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3500457699 0 0) 0x7f3e18003520 con 0x7f3e300a98f0 2024-03-25T19:32:52.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 1 -- 172.21.15.52:0/1765542647 >> v1:172.21.15.52:6790/0 conn(0x7f3e3013f710 legacy=0x7f3e301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:52.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 1 -- 172.21.15.52:0/1765542647 >> v1:172.21.15.179:6789/0 conn(0x7f3e301430c0 legacy=0x7f3e3014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:52.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:32:52.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 1 monclient: found mon.noname-a 2024-03-25T19:32:52.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:52.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:32:52.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 1 -- 172.21.15.52:0/1765542647 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e300b44c0 con 0x7f3e300a98f0 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 10 monclient: _finish_auth 0 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 1 -- 172.21.15.52:0/1765542647 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f3e20001690 con 0x7f3e300a98f0 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e34b48640 5 monclient: authenticate success, global_id 4464 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e34b48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e2d59a640 10 monclient: dump: 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:52.463 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:52.464 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:52.464 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:52.464 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:52.464 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:52.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.462+0000 7f3e34b48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:52.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.481+0000 7f3e2d59a640 1 -- 172.21.15.52:0/1765542647 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e20001870 con 0x7f3e300a98f0 2024-03-25T19:32:52.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e2d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:52.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e2d59a640 1 -- 172.21.15.52:0/1765542647 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f3e20001c80 con 0x7f3e300a98f0 2024-03-25T19:32:52.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e34b48640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:52.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e34b48640 4 set_mon_vals no callback set 2024-03-25T19:32:52.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e2d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:52.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e2d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:52.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e2d59a640 10 monclient: dump: 2024-03-25T19:32:52.482 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:52.482 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:52.482 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.482 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.482 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:52.482 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:52.482 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:52.482 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:52.482 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:52.482 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:52.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e34b48640 10 monclient: shutdown 2024-03-25T19:32:52.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e34b48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:52.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e34b48640 1 -- 172.21.15.52:0/1765542647 >> v1:172.21.15.52:6789/0 conn(0x7f3e300a98f0 legacy=0x7f3e300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:52.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e34b48640 1 -- 172.21.15.52:0/1765542647 shutdown_connections 2024-03-25T19:32:52.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e34b48640 1 -- 172.21.15.52:0/1765542647 >> 172.21.15.52:0/1765542647 conn(0x7f3e300b7ea0 msgr2=0x7f3e300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:52.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e34b48640 1 -- 172.21.15.52:0/1765542647 shutdown_connections 2024-03-25T19:32:52.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.482+0000 7f3e34b48640 1 -- 172.21.15.52:0/1765542647 wait complete. 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.484+0000 7f3e34b48640 10 monclient: build_initial_monmap 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.484+0000 7f3e34b48640 10 monclient: monmap: 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:32:52.484 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:52.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.484+0000 7f3e34b48640 1 librados: starting msgr at 2024-03-25T19:32:52.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.484+0000 7f3e34b48640 1 librados: starting objecter 2024-03-25T19:32:52.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.484+0000 7f3e34b48640 1 Processor -- start 2024-03-25T19:32:52.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.485+0000 7f3e34b48640 1 -- start start 2024-03-25T19:32:52.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.485+0000 7f3e34b48640 1 librados: setting wanted keys 2024-03-25T19:32:52.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.485+0000 7f3e34b48640 1 librados: calling monclient init 2024-03-25T19:32:52.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.485+0000 7f3e34b48640 10 monclient: init 2024-03-25T19:32:52.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.485+0000 7f3e34b48640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:52.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.485+0000 7f3e34b48640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:52.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.486+0000 7f3e34b48640 10 monclient(hunting): picked mon.noname-c con 0x7f3e300a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:32:52.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.486+0000 7f3e34b48640 10 monclient(hunting): picked mon.noname-a con 0x7f3e3013f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:32:52.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.486+0000 7f3e34b48640 10 monclient(hunting): picked mon.noname-b con 0x7f3e301430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:32:52.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.486+0000 7f3e34b48640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e301d51f0 con 0x7f3e3013f710 2024-03-25T19:32:52.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.486+0000 7f3e34b48640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e30137990 con 0x7f3e301430c0 2024-03-25T19:32:52.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.486+0000 7f3e34b48640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e30139990 con 0x7f3e300a98f0 2024-03-25T19:32:52.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.486+0000 7f3e34b48640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:52.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.486+0000 7f3e34b48640 10 monclient(hunting): authenticate will time out at 6220.665995s 2024-03-25T19:32:52.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.498+0000 7f3e2dd9b640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f3e3013f710 0x7f3e301dad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:42694/0 (socket says 172.21.15.52:42694) 2024-03-25T19:32:52.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.498+0000 7f3e2dd9b640 1 -- 172.21.15.52:0/2414407646 learned_addr learned my addr 172.21.15.52:0/2414407646 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:32:52.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.500+0000 7f3e1effd640 10 client.?.objecter ms_handle_connect 0x7f3e3013f710 2024-03-25T19:32:52.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.500+0000 7f3e1effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:52.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.505+0000 7f3e1effd640 10 client.?.objecter ms_handle_connect 0x7f3e301430c0 2024-03-25T19:32:52.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.505+0000 7f3e1effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:52.504 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.505+0000 7f3e1effd640 10 client.?.objecter ms_handle_connect 0x7f3e300a98f0 2024-03-25T19:32:52.505 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.505+0000 7f3e1effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:52.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.511+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548740701 0 0) 0x7f3e30137990 con 0x7f3e301430c0 2024-03-25T19:32:52.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.511+0000 7f3e1effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:52.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.511+0000 7f3e1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:52.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.511+0000 7f3e1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:52.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.511+0000 7f3e1effd640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:32:52.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.511+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e10004eb0 con 0x7f3e301430c0 2024-03-25T19:32:52.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.515+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792187696 0 0) 0x7f3e30139990 con 0x7f3e300a98f0 2024-03-25T19:32:52.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.515+0000 7f3e1effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:52.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.515+0000 7f3e1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:52.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.515+0000 7f3e1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:52.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.515+0000 7f3e1effd640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:32:52.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.515+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e30137990 con 0x7f3e300a98f0 2024-03-25T19:32:52.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.519+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246417985 0 0) 0x7f3e301d51f0 con 0x7f3e3013f710 2024-03-25T19:32:52.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.519+0000 7f3e1effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:52.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.519+0000 7f3e1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:52.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.519+0000 7f3e1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:52.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.519+0000 7f3e1effd640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:32:52.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.519+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e30139990 con 0x7f3e3013f710 2024-03-25T19:32:52.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.533+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1886080353 0 0) 0x7f3e30139990 con 0x7f3e3013f710 2024-03-25T19:32:52.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.533+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e301d51f0 con 0x7f3e3013f710 2024-03-25T19:32:52.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.533+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f3e240019b0 con 0x7f3e3013f710 2024-03-25T19:32:52.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.533+0000 7f3e1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:52.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.533+0000 7f3e1effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:52.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.533+0000 7f3e1effd640 10 monclient(hunting): dump: 2024-03-25T19:32:52.533 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:52.533 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:52.533 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.533 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.533 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:52.534 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:52.534 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:52.534 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:52.534 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:52.534 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:52.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.533+0000 7f3e1effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:52.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.534+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 465132815 0 0) 0x7f3e10004eb0 con 0x7f3e301430c0 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.534+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e30139990 con 0x7f3e301430c0 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.535+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f3e28001b20 con 0x7f3e301430c0 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.535+0000 7f3e1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.535+0000 7f3e1effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.535+0000 7f3e1effd640 10 monclient(hunting): dump: 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:52.535 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:52.536 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:52.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.536+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2282515267 0 0) 0x7f3e30137990 con 0x7f3e300a98f0 2024-03-25T19:32:52.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.536+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e10004eb0 con 0x7f3e300a98f0 2024-03-25T19:32:52.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.536+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f3e20001b80 con 0x7f3e300a98f0 2024-03-25T19:32:52.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.536+0000 7f3e1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:52.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.536+0000 7f3e1effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:52.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.536+0000 7f3e1effd640 10 monclient(hunting): dump: 2024-03-25T19:32:52.537 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:52.537 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:52.537 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.537 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.537 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:52.537 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:52.537 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:52.537 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:52.537 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:52.537 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:52.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1780344043 0 0) 0x7f3e301d51f0 con 0x7f3e3013f710 2024-03-25T19:32:52.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 >> v1:172.21.15.52:6790/0 conn(0x7f3e300a98f0 legacy=0x7f3e30137eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:52.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 >> v1:172.21.15.179:6789/0 conn(0x7f3e301430c0 legacy=0x7f3e301db410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:52.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e1effd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:52.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e1effd640 1 monclient: found mon.a 2024-03-25T19:32:52.543 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e1effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e301385c0 con 0x7f3e3013f710 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e1effd640 10 monclient: _finish_auth 0 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e1effd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e34b48640 5 monclient: authenticate success, global_id 4467 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e34b48640 10 monclient: _renew_subs 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e34b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e34b48640 1 -- 172.21.15.52:0/2414407646 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e30138810 con 0x7f3e3013f710 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e34b48640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e34b48640 10 monclient: _renew_subs 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e34b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e34b48640 1 -- 172.21.15.52:0/2414407646 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e301e3000 con 0x7f3e3013f710 2024-03-25T19:32:52.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.543+0000 7f3e34b48640 1 librados: init done 2024-03-25T19:32:52.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.544+0000 7f3e34b48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:52.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.544+0000 7f3e34b48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:52.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.544+0000 7f3e34b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:52.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.544+0000 7f3e34b48640 1 -- 172.21.15.52:0/2414407646 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e300627b0 con 0x7f3e3013f710 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.852+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e24003610 con 0x7f3e3013f710 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.852+0000 7f3e1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.852+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f3e24003a20 con 0x7f3e3013f710 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.852+0000 7f3e1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.852+0000 7f3e1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.852+0000 7f3e1effd640 10 monclient: dump: 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:32:52.852 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:32:52.853 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:32:52.853 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:32:52.853 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:32:52.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.852+0000 7f3e2cd99640 4 set_mon_vals no callback set 2024-03-25T19:32:52.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.853+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f3e24003c40 con 0x7f3e3013f710 2024-03-25T19:32:52.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.853+0000 7f3e2e59c640 10 monclient: get_auth_request con 0x7f3e1006e0f0 auth_method 0 2024-03-25T19:32:52.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.854+0000 7f3e1effd640 10 client.4467.objecter ms_handle_connect 0x7f3e1006e0f0 2024-03-25T19:32:52.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.859+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f3e24054700 con 0x7f3e3013f710 2024-03-25T19:32:52.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.859+0000 7f3e1effd640 10 client.4467.objecter ms_dispatch 0x7f3e30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:52.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.859+0000 7f3e1effd640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:52.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.859+0000 7f3e1effd640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:52.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.859+0000 7f3e1effd640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-25T19:32:52.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.859+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e24050210 con 0x7f3e3013f710 2024-03-25T19:32:52.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.859+0000 7f3e1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:52.860 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:52.859+0000 7f3e1effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:53.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.085+0000 7f3e34b48640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:53.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.085+0000 7f3e34b48640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:53.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.085+0000 7f3e34b48640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:32:53.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.085+0000 7f3e34b48640 1 -- 172.21.15.52:0/2414407646 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e30137b20 con 0x7f3e3013f710 2024-03-25T19:32:53.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.136+0000 7f3e1effd640 1 -- 172.21.15.52:0/2414407646 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f3e24052140 con 0x7f3e3013f710 2024-03-25T19:32:53.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.136+0000 7f3e1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:53.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.136+0000 7f3e1effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:53.136 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:32:53.136 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:53.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.138+0000 7f3e34b48640 10 librados: watch_flush enter 2024-03-25T19:32:53.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.138+0000 7f3e34b48640 10 librados: watch_flush exit 2024-03-25T19:32:53.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.138+0000 7f3e34b48640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-25T19:32:53.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.138+0000 7f3e34b48640 10 client.4467.objecter successfully canceled tick 2024-03-25T19:32:53.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.139+0000 7f3e34b48640 1 -- 172.21.15.52:0/2414407646 >> v1:172.21.15.52:6800/3967427147 conn(0x7f3e1006e0f0 legacy=0x7f3e100705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:53.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.139+0000 7f3e34b48640 10 monclient: shutdown 2024-03-25T19:32:53.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.139+0000 7f3e34b48640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:53.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.139+0000 7f3e34b48640 1 -- 172.21.15.52:0/2414407646 >> v1:172.21.15.52:6789/0 conn(0x7f3e3013f710 legacy=0x7f3e301dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:53.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.139+0000 7f3e34b48640 1 -- 172.21.15.52:0/2414407646 shutdown_connections 2024-03-25T19:32:53.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.139+0000 7f3e34b48640 1 -- 172.21.15.52:0/2414407646 >> 172.21.15.52:0/2414407646 conn(0x7f3e300b7ea0 msgr2=0x7f3e3013c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:53.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.139+0000 7f3e34b48640 1 -- 172.21.15.52:0/2414407646 shutdown_connections 2024-03-25T19:32:53.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.139+0000 7f3e34b48640 1 -- 172.21.15.52:0/2414407646 wait complete. 2024-03-25T19:32:53.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:32:53.139+0000 7f3e34b48640 1 librados: shutdown 2024-03-25T19:32:53.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:59.955 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:00.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.072+0000 7fa184532640 10 monclient: get_monmap_and_config 2024-03-25T19:33:00.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.072+0000 7fa184532640 10 monclient: build_initial_monmap 2024-03-25T19:33:00.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.072+0000 7fa184532640 10 monclient: monmap: 2024-03-25T19:33:00.072 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:00.072 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:00.072 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:33:00.072651+0000 2024-03-25T19:33:00.072 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:33:00.072651+0000 2024-03-25T19:33:00.072 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:00.072 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:00.072 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:00.072 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:33:00.072 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:33:00.072 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:00.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.072+0000 7fa184532640 1 Processor -- start 2024-03-25T19:33:00.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.072+0000 7fa184532640 1 -- start start 2024-03-25T19:33:00.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.072+0000 7fa184532640 10 monclient: init 2024-03-25T19:33:00.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.073+0000 7fa184532640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:00.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.073+0000 7fa184532640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:00.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.073+0000 7fa184532640 10 monclient(hunting): picked mon.noname-b con 0x7fa17c0b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:00.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.073+0000 7fa184532640 10 monclient(hunting): picked mon.noname-c con 0x7fa17c0b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:00.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.073+0000 7fa184532640 10 monclient(hunting): picked mon.noname-a con 0x7fa17c14be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:00.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.073+0000 7fa184532640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c0af940 con 0x7fa17c14be90 2024-03-25T19:33:00.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.073+0000 7fa184532640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c0b0a70 con 0x7fa17c0b1730 2024-03-25T19:33:00.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.073+0000 7fa184532640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c0b0d50 con 0x7fa17c0b4d60 2024-03-25T19:33:00.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.073+0000 7fa184532640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:00.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.073+0000 7fa184532640 10 monclient(hunting): authenticate will time out at 6228.253615s 2024-03-25T19:33:00.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.081+0000 7fa182acf640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fa17c14be90 0x7fa17c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:59910/0 (socket says 172.21.15.52:59910) 2024-03-25T19:33:00.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.081+0000 7fa182acf640 1 -- 172.21.15.52:0/499526420 learned_addr learned my addr 172.21.15.52:0/499526420 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:00.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790236273 0 0) 0x7fa17c0b0a70 con 0x7fa17c0b1730 2024-03-25T19:33:00.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:00.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:00.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:00.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:33:00.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa168003520 con 0x7fa17c0b1730 2024-03-25T19:33:00.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944131662 0 0) 0x7fa17c0b0d50 con 0x7fa17c0b4d60 2024-03-25T19:33:00.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:00.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:00.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:00.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:33:00.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.104+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c0b0a70 con 0x7fa17c0b4d60 2024-03-25T19:33:00.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.105+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780750940 0 0) 0x7fa17c0af940 con 0x7fa17c14be90 2024-03-25T19:33:00.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.105+0000 7fa1812cc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:00.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.105+0000 7fa1812cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:00.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.105+0000 7fa1812cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:00.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.105+0000 7fa1812cc640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:33:00.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.105+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c0b0d50 con 0x7fa17c14be90 2024-03-25T19:33:00.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1574472150 0 0) 0x7fa17c0b0a70 con 0x7fa17c0b4d60 2024-03-25T19:33:00.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 >> v1:172.21.15.179:6789/0 conn(0x7fa17c0b1730 legacy=0x7fa17c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 >> v1:172.21.15.52:6789/0 conn(0x7fa17c14be90 legacy=0x7fa17c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 10 monclient: _finish_hunting 0 2024-03-25T19:33:00.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 1 monclient: found mon.noname-b 2024-03-25T19:33:00.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:00.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c0b1030 con 0x7fa17c0b4d60 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 10 monclient: _finish_auth 0 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 10 monclient: _check_auth_tickets 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa178001670 con 0x7fa17c0b4d60 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa1812cc640 10 monclient: dump: 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:00.115 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:00.116 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:00.116 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:00.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa184532640 5 monclient: authenticate success, global_id 4466 2024-03-25T19:33:00.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.114+0000 7fa184532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:00.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.136+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1780017d0 con 0x7fa17c0b4d60 2024-03-25T19:33:00.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.136+0000 7fa1812cc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:00.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.136+0000 7fa1812cc640 1 -- 172.21.15.52:0/499526420 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa178001be0 con 0x7fa17c0b4d60 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.136+0000 7fa184532640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.136+0000 7fa184532640 4 set_mon_vals no callback set 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.136+0000 7fa1812cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.136+0000 7fa1812cc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.136+0000 7fa1812cc640 10 monclient: dump: 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:00.136 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:00.137 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:00.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.136+0000 7fa184532640 10 monclient: shutdown 2024-03-25T19:33:00.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.136+0000 7fa184532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:00.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.136+0000 7fa184532640 1 -- 172.21.15.52:0/499526420 >> v1:172.21.15.52:6790/0 conn(0x7fa17c0b4d60 legacy=0x7fa17c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.137+0000 7fa184532640 1 -- 172.21.15.52:0/499526420 shutdown_connections 2024-03-25T19:33:00.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.137+0000 7fa184532640 1 -- 172.21.15.52:0/499526420 >> 172.21.15.52:0/499526420 conn(0x7fa17c0617a0 msgr2=0x7fa17c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:00.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.137+0000 7fa184532640 1 -- 172.21.15.52:0/499526420 shutdown_connections 2024-03-25T19:33:00.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.137+0000 7fa184532640 1 -- 172.21.15.52:0/499526420 wait complete. 2024-03-25T19:33:00.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.137+0000 7fa184532640 10 monclient: build_initial_monmap 2024-03-25T19:33:00.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.137+0000 7fa184532640 10 monclient: monmap: 2024-03-25T19:33:00.139 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:00.139 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.138+0000 7fa184532640 1 librados: starting msgr at 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.138+0000 7fa184532640 1 librados: starting objecter 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.138+0000 7fa184532640 1 Processor -- start 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.138+0000 7fa184532640 1 -- start start 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.138+0000 7fa184532640 1 librados: setting wanted keys 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.138+0000 7fa184532640 1 librados: calling monclient init 2024-03-25T19:33:00.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.138+0000 7fa184532640 10 monclient: init 2024-03-25T19:33:00.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.139+0000 7fa184532640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:00.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.139+0000 7fa184532640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:00.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.139+0000 7fa184532640 10 monclient(hunting): picked mon.noname-c con 0x7fa17c0b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:00.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.139+0000 7fa184532640 10 monclient(hunting): picked mon.noname-b con 0x7fa17c0b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:00.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.139+0000 7fa184532640 10 monclient(hunting): picked mon.noname-a con 0x7fa17c14be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:00.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.139+0000 7fa184532640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c1dabc0 con 0x7fa17c14be90 2024-03-25T19:33:00.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.139+0000 7fa184532640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c140e50 con 0x7fa17c0b4d60 2024-03-25T19:33:00.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.139+0000 7fa184532640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17c143b40 con 0x7fa17c0b1730 2024-03-25T19:33:00.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.139+0000 7fa184532640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:00.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.139+0000 7fa184532640 10 monclient(hunting): authenticate will time out at 6228.319489s 2024-03-25T19:33:00.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.149+0000 7fa182acf640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fa17c14be90 0x7fa17c1e2ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:59958/0 (socket says 172.21.15.52:59958) 2024-03-25T19:33:00.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.149+0000 7fa182acf640 1 -- 172.21.15.52:0/466804855 learned_addr learned my addr 172.21.15.52:0/466804855 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:00.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.153+0000 7fa172ffd640 10 client.?.objecter ms_handle_connect 0x7fa17c14be90 2024-03-25T19:33:00.152 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.153+0000 7fa172ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:00.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.155+0000 7fa172ffd640 10 client.?.objecter ms_handle_connect 0x7fa17c0b4d60 2024-03-25T19:33:00.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.155+0000 7fa172ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:00.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.157+0000 7fa172ffd640 10 client.?.objecter ms_handle_connect 0x7fa17c0b1730 2024-03-25T19:33:00.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.157+0000 7fa172ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:00.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.159+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326586267 0 0) 0x7fa17c1dabc0 con 0x7fa17c14be90 2024-03-25T19:33:00.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.159+0000 7fa172ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:00.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.159+0000 7fa172ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:00.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.159+0000 7fa172ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:00.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.159+0000 7fa172ffd640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:33:00.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.159+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa160004eb0 con 0x7fa17c14be90 2024-03-25T19:33:00.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.170+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274975224 0 0) 0x7fa17c140e50 con 0x7fa17c0b4d60 2024-03-25T19:33:00.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.170+0000 7fa172ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:00.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.170+0000 7fa172ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:00.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.170+0000 7fa172ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:00.169 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.170+0000 7fa172ffd640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:33:00.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.170+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c1dabc0 con 0x7fa17c0b4d60 2024-03-25T19:33:00.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.172+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951227018 0 0) 0x7fa17c143b40 con 0x7fa17c0b1730 2024-03-25T19:33:00.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.172+0000 7fa172ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:00.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.172+0000 7fa172ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:00.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.172+0000 7fa172ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:00.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.172+0000 7fa172ffd640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:33:00.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.172+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c140e50 con 0x7fa17c0b1730 2024-03-25T19:33:00.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.184+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 9457204 0 0) 0x7fa160004eb0 con 0x7fa17c14be90 2024-03-25T19:33:00.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.184+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa17c143b40 con 0x7fa17c14be90 2024-03-25T19:33:00.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.184+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa174001bd0 con 0x7fa17c14be90 2024-03-25T19:33:00.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.184+0000 7fa172ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:00.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.184+0000 7fa172ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:00.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.184+0000 7fa172ffd640 10 monclient(hunting): dump: 2024-03-25T19:33:00.184 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:00.184 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:00.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.184+0000 7fa172ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:00.189 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1155510352 0 0) 0x7fa17c143b40 con 0x7fa17c14be90 2024-03-25T19:33:00.189 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 >> v1:172.21.15.52:6790/0 conn(0x7fa17c0b1730 legacy=0x7fa17c1db0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 >> v1:172.21.15.179:6789/0 conn(0x7fa17c0b4d60 legacy=0x7fa17c130d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa172ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa172ffd640 1 monclient: found mon.a 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa172ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa172ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c143cd0 con 0x7fa17c14be90 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa172ffd640 10 monclient: _finish_auth 0 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa172ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa172ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa184532640 5 monclient: authenticate success, global_id 4473 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa184532640 10 monclient: _renew_subs 2024-03-25T19:33:00.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa184532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:33:00.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.189+0000 7fa184532640 1 -- 172.21.15.52:0/466804855 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17c136b40 con 0x7fa17c14be90 2024-03-25T19:33:00.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.190+0000 7fa184532640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:00.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.190+0000 7fa184532640 10 monclient: _renew_subs 2024-03-25T19:33:00.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.190+0000 7fa184532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:33:00.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.190+0000 7fa184532640 1 -- 172.21.15.52:0/466804855 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17c1db990 con 0x7fa17c14be90 2024-03-25T19:33:00.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.190+0000 7fa184532640 1 librados: init done 2024-03-25T19:33:00.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.190+0000 7fa184532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:00.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.191+0000 7fa184532640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:00.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.191+0000 7fa184532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:33:00.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.191+0000 7fa184532640 1 -- 172.21.15.52:0/466804855 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17c140cb0 con 0x7fa17c14be90 2024-03-25T19:33:00.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.215+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa174002080 con 0x7fa17c14be90 2024-03-25T19:33:00.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.215+0000 7fa172ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:00.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.215+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa174004b60 con 0x7fa17c14be90 2024-03-25T19:33:00.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.215+0000 7fa172ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.215+0000 7fa172ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.215+0000 7fa172ffd640 10 monclient: dump: 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:00.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.215+0000 7fa180acb640 4 set_mon_vals no callback set 2024-03-25T19:33:00.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.216+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fa174004d80 con 0x7fa17c14be90 2024-03-25T19:33:00.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.216+0000 7fa1822ce640 10 monclient: get_auth_request con 0x7fa16006dd00 auth_method 0 2024-03-25T19:33:00.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.216+0000 7fa172ffd640 10 client.4473.objecter ms_handle_connect 0x7fa16006dd00 2024-03-25T19:33:00.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.518+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fa174054770 con 0x7fa17c14be90 2024-03-25T19:33:00.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.518+0000 7fa172ffd640 10 client.4473.objecter ms_dispatch 0x7fa17c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:00.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.518+0000 7fa172ffd640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:00.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.518+0000 7fa172ffd640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:00.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.518+0000 7fa172ffd640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T19:33:00.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.518+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1740500b0 con 0x7fa17c14be90 2024-03-25T19:33:00.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.518+0000 7fa172ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:00.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.518+0000 7fa172ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:00.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.759+0000 7fa184532640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:00.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.759+0000 7fa184532640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:00.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.759+0000 7fa184532640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:33:00.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.759+0000 7fa184532640 1 -- 172.21.15.52:0/466804855 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa17c143cd0 con 0x7fa17c14be90 2024-03-25T19:33:00.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.793+0000 7fa172ffd640 1 -- 172.21.15.52:0/466804855 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fa17c143cd0 con 0x7fa17c14be90 2024-03-25T19:33:00.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.793+0000 7fa172ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:00.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.793+0000 7fa172ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:00.793 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:33:00.793 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:00.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.795+0000 7fa184532640 10 librados: watch_flush enter 2024-03-25T19:33:00.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.795+0000 7fa184532640 10 librados: watch_flush exit 2024-03-25T19:33:00.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.795+0000 7fa184532640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T19:33:00.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.795+0000 7fa184532640 10 client.4473.objecter successfully canceled tick 2024-03-25T19:33:00.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.795+0000 7fa184532640 1 -- 172.21.15.52:0/466804855 >> v1:172.21.15.52:6800/3967427147 conn(0x7fa16006dd00 legacy=0x7fa1600701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.795+0000 7fa184532640 10 monclient: shutdown 2024-03-25T19:33:00.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.795+0000 7fa184532640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:00.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.795+0000 7fa184532640 1 -- 172.21.15.52:0/466804855 >> v1:172.21.15.52:6789/0 conn(0x7fa17c14be90 legacy=0x7fa17c1e2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:00.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.796+0000 7fa184532640 1 -- 172.21.15.52:0/466804855 shutdown_connections 2024-03-25T19:33:00.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.796+0000 7fa184532640 1 -- 172.21.15.52:0/466804855 >> 172.21.15.52:0/466804855 conn(0x7fa17c0617a0 msgr2=0x7fa17c13c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:00.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.796+0000 7fa184532640 1 -- 172.21.15.52:0/466804855 shutdown_connections 2024-03-25T19:33:00.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.796+0000 7fa184532640 1 -- 172.21.15.52:0/466804855 wait complete. 2024-03-25T19:33:00.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:00.796+0000 7fa184532640 1 librados: shutdown 2024-03-25T19:33:00.806 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:07.612 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.731+0000 7f201eb3a640 10 monclient: get_monmap_and_config 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.731+0000 7f201eb3a640 10 monclient: build_initial_monmap 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.731+0000 7f201eb3a640 10 monclient: monmap: 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:33:07.732154+0000 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:33:07.732154+0000 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:33:07.731 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:07.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.732+0000 7f201eb3a640 1 Processor -- start 2024-03-25T19:33:07.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.732+0000 7f201eb3a640 1 -- start start 2024-03-25T19:33:07.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.732+0000 7f201eb3a640 10 monclient: init 2024-03-25T19:33:07.733 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.733+0000 7f201eb3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:07.733 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.733+0000 7f201eb3a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:07.733 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.733+0000 7f201eb3a640 10 monclient(hunting): picked mon.noname-b con 0x7f20180a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:07.733 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.733+0000 7f201eb3a640 10 monclient(hunting): picked mon.noname-a con 0x7f201813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:07.733 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.733+0000 7f201eb3a640 10 monclient(hunting): picked mon.noname-c con 0x7f20181430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:07.733 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.733+0000 7f201eb3a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2018142520 con 0x7f201813f710 2024-03-25T19:33:07.733 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.733+0000 7f201eb3a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20180b28a0 con 0x7f20181430c0 2024-03-25T19:33:07.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.733+0000 7f201eb3a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20180b2a30 con 0x7f20180a98f0 2024-03-25T19:33:07.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.733+0000 7f201eb3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:07.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.733+0000 7f201eb3a640 10 monclient(hunting): authenticate will time out at 6235.913677s 2024-03-25T19:33:07.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.744+0000 7f201d0d7640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f20181430c0 0x7f201814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:55260/0 (socket says 172.21.15.52:55260) 2024-03-25T19:33:07.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.744+0000 7f201d0d7640 1 -- 172.21.15.52:0/14226859 learned_addr learned my addr 172.21.15.52:0/14226859 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:07.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.756+0000 7f20177fe640 1 -- 172.21.15.52:0/14226859 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284129991 0 0) 0x7f20180b28a0 con 0x7f20181430c0 2024-03-25T19:33:07.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.756+0000 7f20177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:07.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.756+0000 7f20177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:07.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.756+0000 7f20177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:07.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.756+0000 7f20177fe640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:33:07.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.756+0000 7f20177fe640 1 -- 172.21.15.52:0/14226859 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2008003520 con 0x7f20181430c0 2024-03-25T19:33:07.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.773+0000 7f20177fe640 1 -- 172.21.15.52:0/14226859 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91782623 0 0) 0x7f2018142520 con 0x7f201813f710 2024-03-25T19:33:07.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.773+0000 7f20177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:07.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.773+0000 7f20177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:07.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.773+0000 7f20177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:07.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.773+0000 7f20177fe640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:33:07.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.773+0000 7f20177fe640 1 -- 172.21.15.52:0/14226859 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20180b28a0 con 0x7f201813f710 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 1 -- 172.21.15.52:0/14226859 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1906034506 0 0) 0x7f2008003520 con 0x7f20181430c0 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 1 -- 172.21.15.52:0/14226859 >> v1:172.21.15.52:6790/0 conn(0x7f20180a98f0 legacy=0x7f20180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 1 -- 172.21.15.52:0/14226859 >> v1:172.21.15.52:6789/0 conn(0x7f201813f710 legacy=0x7f20181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 1 monclient: found mon.noname-c 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 1 -- 172.21.15.52:0/14226859 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20180b44c0 con 0x7f20181430c0 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 1 -- 172.21.15.52:0/14226859 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2010001a10 con 0x7f20181430c0 2024-03-25T19:33:07.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f201eb3a640 5 monclient: authenticate success, global_id 4483 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f201eb3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f20177fe640 10 monclient: dump: 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:07.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.774+0000 7f201eb3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.778+0000 7f20177fe640 1 -- 172.21.15.52:0/14226859 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2010001b70 con 0x7f20181430c0 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.778+0000 7f20177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.778+0000 7f20177fe640 1 -- 172.21.15.52:0/14226859 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2010001f80 con 0x7f20181430c0 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.778+0000 7f201eb3a640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.778+0000 7f201eb3a640 4 set_mon_vals no callback set 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.778+0000 7f20177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.778+0000 7f20177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.778+0000 7f20177fe640 10 monclient: dump: 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:07.778 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:07.779 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:07.779 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:07.779 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:07.779 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:07.779 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:07.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.779+0000 7f201eb3a640 10 monclient: shutdown 2024-03-25T19:33:07.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.779+0000 7f201eb3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:07.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.779+0000 7f201eb3a640 1 -- 172.21.15.52:0/14226859 >> v1:172.21.15.179:6789/0 conn(0x7f20181430c0 legacy=0x7f201814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.779+0000 7f201eb3a640 1 -- 172.21.15.52:0/14226859 shutdown_connections 2024-03-25T19:33:07.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.779+0000 7f201eb3a640 1 -- 172.21.15.52:0/14226859 >> 172.21.15.52:0/14226859 conn(0x7f20180b7ea0 msgr2=0x7f20180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:07.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.779+0000 7f201eb3a640 1 -- 172.21.15.52:0/14226859 shutdown_connections 2024-03-25T19:33:07.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.780+0000 7f201eb3a640 1 -- 172.21.15.52:0/14226859 wait complete. 2024-03-25T19:33:07.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.780+0000 7f201eb3a640 10 monclient: build_initial_monmap 2024-03-25T19:33:07.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.780+0000 7f201eb3a640 10 monclient: monmap: 2024-03-25T19:33:07.780 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:07.780 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:07.781 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:33:07.781 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:33:07.781 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:07.781 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:33:07.781 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:07.781 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:33:07.781 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:33:07.781 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:07.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.781+0000 7f201eb3a640 1 librados: starting msgr at 2024-03-25T19:33:07.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.781+0000 7f201eb3a640 1 librados: starting objecter 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.781+0000 7f201eb3a640 1 Processor -- start 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.781+0000 7f201eb3a640 1 -- start start 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.781+0000 7f201eb3a640 1 librados: setting wanted keys 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.781+0000 7f201eb3a640 1 librados: calling monclient init 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.781+0000 7f201eb3a640 10 monclient: init 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.782+0000 7f201eb3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.782+0000 7f201eb3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.782+0000 7f201eb3a640 10 monclient(hunting): picked mon.noname-a con 0x7f20180a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.782+0000 7f201eb3a640 10 monclient(hunting): picked mon.noname-c con 0x7f201813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.782+0000 7f201eb3a640 10 monclient(hunting): picked mon.noname-b con 0x7f20181430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.782+0000 7f201eb3a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2018130e90 con 0x7f20180a98f0 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.782+0000 7f201eb3a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2018136cc0 con 0x7f20181430c0 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.782+0000 7f201eb3a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2018136fa0 con 0x7f201813f710 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.782+0000 7f201eb3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:07.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.782+0000 7f201eb3a640 10 monclient(hunting): authenticate will time out at 6235.962411s 2024-03-25T19:33:07.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.791+0000 7f201d0d7640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f20181430c0 0x7f2018137980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:55294/0 (socket says 172.21.15.52:55294) 2024-03-25T19:33:07.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.791+0000 7f201d0d7640 1 -- 172.21.15.52:0/1862529040 learned_addr learned my addr 172.21.15.52:0/1862529040 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:07.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.793+0000 7f20157fa640 10 client.?.objecter ms_handle_connect 0x7f20181430c0 2024-03-25T19:33:07.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.793+0000 7f20157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:07.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.795+0000 7f20157fa640 10 client.?.objecter ms_handle_connect 0x7f201813f710 2024-03-25T19:33:07.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.795+0000 7f20157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:07.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.808+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026748712 0 0) 0x7f2018136cc0 con 0x7f20181430c0 2024-03-25T19:33:07.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.808+0000 7f20157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:07.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.808+0000 7f20157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:07.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.808+0000 7f20157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:07.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.808+0000 7f20157fa640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:33:07.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.808+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ffc004eb0 con 0x7f20181430c0 2024-03-25T19:33:07.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.811+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806289983 0 0) 0x7f2018136fa0 con 0x7f201813f710 2024-03-25T19:33:07.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.811+0000 7f20157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:07.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.811+0000 7f20157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:07.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.811+0000 7f20157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:07.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.811+0000 7f20157fa640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:33:07.811 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.811+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2018136cc0 con 0x7f201813f710 2024-03-25T19:33:07.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.821+0000 7f20157fa640 10 client.?.objecter ms_handle_connect 0x7f20180a98f0 2024-03-25T19:33:07.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.821+0000 7f20157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560463880 0 0) 0x7f2018130e90 con 0x7f20180a98f0 2024-03-25T19:33:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:33:07.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2018136fa0 con 0x7f20180a98f0 2024-03-25T19:33:07.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3902239751 0 0) 0x7f1ffc004eb0 con 0x7f20181430c0 2024-03-25T19:33:07.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2018130e90 con 0x7f20181430c0 2024-03-25T19:33:07.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2010001ec0 con 0x7f20181430c0 2024-03-25T19:33:07.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:07.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:07.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 10 monclient(hunting): dump: 2024-03-25T19:33:07.825 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:07.825 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:07.825 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:07.825 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:07.826 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:07.826 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:07.826 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:07.826 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:07.826 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:07.826 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:07.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.825+0000 7f20157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 652527891 0 0) 0x7f2018130e90 con 0x7f20181430c0 2024-03-25T19:33:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 >> v1:172.21.15.52:6790/0 conn(0x7f201813f710 legacy=0x7f201813b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 >> v1:172.21.15.52:6789/0 conn(0x7f20180a98f0 legacy=0x7f2018135300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f20157fa640 10 monclient: _finish_hunting 0 2024-03-25T19:33:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f20157fa640 1 monclient: found mon.b 2024-03-25T19:33:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f20157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f20157fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2018137280 con 0x7f20181430c0 2024-03-25T19:33:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f20157fa640 10 monclient: _finish_auth 0 2024-03-25T19:33:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f20157fa640 10 monclient: _check_auth_tickets 2024-03-25T19:33:07.835 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f20157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:07.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f201eb3a640 5 monclient: authenticate success, global_id 4486 2024-03-25T19:33:07.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f201eb3a640 10 monclient: _renew_subs 2024-03-25T19:33:07.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f201eb3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:07.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f201eb3a640 1 -- 172.21.15.52:0/1862529040 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2018137560 con 0x7f20181430c0 2024-03-25T19:33:07.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f201eb3a640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:07.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f201eb3a640 10 monclient: _renew_subs 2024-03-25T19:33:07.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f201eb3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:07.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f201eb3a640 1 -- 172.21.15.52:0/1862529040 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20181e36c0 con 0x7f20181430c0 2024-03-25T19:33:07.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.835+0000 7f201eb3a640 1 librados: init done 2024-03-25T19:33:07.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.836+0000 7f201eb3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:07.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.836+0000 7f201eb3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:07.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.836+0000 7f201eb3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:07.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:07.836+0000 7f201eb3a640 1 -- 172.21.15.52:0/1862529040 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20180627b0 con 0x7f20181430c0 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.098+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2010004e10 con 0x7f20181430c0 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.098+0000 7f20157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.098+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f20100015d0 con 0x7f20181430c0 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.098+0000 7f20157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.098+0000 7f20157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.098+0000 7f20157fa640 10 monclient: dump: 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:08.098 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:08.099 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:08.099 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:08.099 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:08.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.098+0000 7f2016ffd640 4 set_mon_vals no callback set 2024-03-25T19:33:08.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.099+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f201001af90 con 0x7f20181430c0 2024-03-25T19:33:08.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.100+0000 7f201c8d6640 10 monclient: get_auth_request con 0x7f1ffc06dda0 auth_method 0 2024-03-25T19:33:08.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.100+0000 7f20157fa640 10 client.4486.objecter ms_handle_connect 0x7f1ffc06dda0 2024-03-25T19:33:08.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.101+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f2010054ba0 con 0x7f20181430c0 2024-03-25T19:33:08.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.101+0000 7f20157fa640 10 client.4486.objecter ms_dispatch 0x7f2018000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:08.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.101+0000 7f20157fa640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:08.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.101+0000 7f20157fa640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:08.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.101+0000 7f20157fa640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-25T19:33:08.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.105+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20100506b0 con 0x7f20181430c0 2024-03-25T19:33:08.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.105+0000 7f20157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:08.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.105+0000 7f20157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:08.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.355+0000 7f201eb3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:08.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.356+0000 7f201eb3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:08.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.356+0000 7f201eb3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:08.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.356+0000 7f201eb3a640 1 -- 172.21.15.52:0/1862529040 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f201813ce70 con 0x7f20181430c0 2024-03-25T19:33:08.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.477+0000 7f20157fa640 1 -- 172.21.15.52:0/1862529040 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f20100525e0 con 0x7f20181430c0 2024-03-25T19:33:08.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.477+0000 7f20157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:08.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.477+0000 7f20157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:33:08.477 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:08.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.479+0000 7f201eb3a640 10 librados: watch_flush enter 2024-03-25T19:33:08.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.479+0000 7f201eb3a640 10 librados: watch_flush exit 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.479+0000 7f201eb3a640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.479+0000 7f201eb3a640 10 client.4486.objecter successfully canceled tick 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.479+0000 7f201eb3a640 1 -- 172.21.15.52:0/1862529040 >> v1:172.21.15.52:6800/3967427147 conn(0x7f1ffc06dda0 legacy=0x7f1ffc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.479+0000 7f201eb3a640 10 monclient: shutdown 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.479+0000 7f201eb3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.479+0000 7f201eb3a640 1 -- 172.21.15.52:0/1862529040 >> v1:172.21.15.179:6789/0 conn(0x7f20181430c0 legacy=0x7f2018137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.480+0000 7f201eb3a640 1 -- 172.21.15.52:0/1862529040 shutdown_connections 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.480+0000 7f201eb3a640 1 -- 172.21.15.52:0/1862529040 >> 172.21.15.52:0/1862529040 conn(0x7f20180b7ea0 msgr2=0x7f2018074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.480+0000 7f201eb3a640 1 -- 172.21.15.52:0/1862529040 shutdown_connections 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.480+0000 7f201eb3a640 1 -- 172.21.15.52:0/1862529040 wait complete. 2024-03-25T19:33:08.480 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:08.480+0000 7f201eb3a640 1 librados: shutdown 2024-03-25T19:33:08.490 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:12.069 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:06:05.668 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.079 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:06:05.678 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.079 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:06:05.621 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.113 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:06:05.711 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.137 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:06:05.679 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.165 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:06:05.707 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.166 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:06:05.765 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.173 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:06:05.772 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:12.220 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:06:05.819 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:15.296 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:15.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.413+0000 7f2d74904640 10 monclient: get_monmap_and_config 2024-03-25T19:33:15.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.413+0000 7f2d74904640 10 monclient: build_initial_monmap 2024-03-25T19:33:15.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.413+0000 7f2d74904640 10 monclient: monmap: 2024-03-25T19:33:15.414 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:15.414 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:15.414 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:33:15.414692+0000 2024-03-25T19:33:15.414 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:33:15.414692+0000 2024-03-25T19:33:15.414 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:15.414 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:15.414 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:15.414 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:33:15.415 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:33:15.415 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:15.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.413+0000 7f2d74904640 1 Processor -- start 2024-03-25T19:33:15.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.413+0000 7f2d74904640 1 -- start start 2024-03-25T19:33:15.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.413+0000 7f2d74904640 10 monclient: init 2024-03-25T19:33:15.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.414+0000 7f2d74904640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:15.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.414+0000 7f2d74904640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:15.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.414+0000 7f2d74904640 10 monclient(hunting): picked mon.noname-a con 0x7f2d700a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:15.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.414+0000 7f2d74904640 10 monclient(hunting): picked mon.noname-c con 0x7f2d7013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:15.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.414+0000 7f2d74904640 10 monclient(hunting): picked mon.noname-b con 0x7f2d701430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:15.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.414+0000 7f2d74904640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d70142520 con 0x7f2d700a98f0 2024-03-25T19:33:15.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.414+0000 7f2d74904640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d700b28a0 con 0x7f2d7013f710 2024-03-25T19:33:15.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.414+0000 7f2d74904640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d700b2a30 con 0x7f2d701430c0 2024-03-25T19:33:15.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.414+0000 7f2d74904640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:15.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.414+0000 7f2d74904640 10 monclient(hunting): authenticate will time out at 6243.595460s 2024-03-25T19:33:15.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.421+0000 7f2d6e59c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f2d700a98f0 0x7f2d700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:40754/0 (socket says 172.21.15.52:40754) 2024-03-25T19:33:15.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.421+0000 7f2d6e59c640 1 -- 172.21.15.52:0/3773216171 learned_addr learned my addr 172.21.15.52:0/3773216171 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:15.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.436+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547219211 0 0) 0x7f2d700b2a30 con 0x7f2d701430c0 2024-03-25T19:33:15.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.436+0000 7f2d6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:15.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.436+0000 7f2d6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:15.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.436+0000 7f2d6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:15.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.436+0000 7f2d6dd9b640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:33:15.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.437+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d5c003520 con 0x7f2d701430c0 2024-03-25T19:33:15.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.439+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465248452 0 0) 0x7f2d70142520 con 0x7f2d700a98f0 2024-03-25T19:33:15.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.439+0000 7f2d6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:15.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.439+0000 7f2d6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:15.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.439+0000 7f2d6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:15.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.439+0000 7f2d6dd9b640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:33:15.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.439+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d700b2a30 con 0x7f2d700a98f0 2024-03-25T19:33:15.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.442+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983607897 0 0) 0x7f2d700b28a0 con 0x7f2d7013f710 2024-03-25T19:33:15.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.442+0000 7f2d6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:15.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.442+0000 7f2d6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:15.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.442+0000 7f2d6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:15.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.442+0000 7f2d6dd9b640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:33:15.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.442+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d70142520 con 0x7f2d7013f710 2024-03-25T19:33:15.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3421628766 0 0) 0x7f2d70142520 con 0x7f2d7013f710 2024-03-25T19:33:15.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 >> v1:172.21.15.52:6790/0 conn(0x7f2d701430c0 legacy=0x7f2d7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:15.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 >> v1:172.21.15.52:6789/0 conn(0x7f2d700a98f0 legacy=0x7f2d700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:15.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:33:15.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 1 monclient: found mon.noname-c 2024-03-25T19:33:15.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:15.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:33:15.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d700b44c0 con 0x7f2d7013f710 2024-03-25T19:33:15.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2d580015b0 con 0x7f2d7013f710 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d6dd9b640 10 monclient: dump: 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:15.450 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:15.451 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:15.451 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:15.451 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:15.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d74904640 5 monclient: authenticate success, global_id 4489 2024-03-25T19:33:15.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.448+0000 7f2d74904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:15.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.461+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d58001710 con 0x7f2d7013f710 2024-03-25T19:33:15.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.461+0000 7f2d6dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:15.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.461+0000 7f2d6dd9b640 1 -- 172.21.15.52:0/3773216171 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2d58001b20 con 0x7f2d7013f710 2024-03-25T19:33:15.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.461+0000 7f2d6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:15.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.461+0000 7f2d6dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:15.462 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.461+0000 7f2d6dd9b640 10 monclient: dump: 2024-03-25T19:33:15.462 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:15.462 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:15.462 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.463 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.463 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:15.463 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:15.463 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:15.463 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:15.463 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:15.463 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:15.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.461+0000 7f2d74904640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:15.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.461+0000 7f2d74904640 4 set_mon_vals no callback set 2024-03-25T19:33:15.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.462+0000 7f2d74904640 10 monclient: shutdown 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.462+0000 7f2d74904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.462+0000 7f2d74904640 1 -- 172.21.15.52:0/3773216171 >> v1:172.21.15.179:6789/0 conn(0x7f2d7013f710 legacy=0x7f2d701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.462+0000 7f2d74904640 1 -- 172.21.15.52:0/3773216171 shutdown_connections 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.462+0000 7f2d74904640 1 -- 172.21.15.52:0/3773216171 >> 172.21.15.52:0/3773216171 conn(0x7f2d700b7ea0 msgr2=0x7f2d700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.462+0000 7f2d74904640 1 -- 172.21.15.52:0/3773216171 shutdown_connections 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.462+0000 7f2d74904640 1 -- 172.21.15.52:0/3773216171 wait complete. 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.463+0000 7f2d74904640 10 monclient: build_initial_monmap 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.463+0000 7f2d74904640 10 monclient: monmap: 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:15.465 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:33:15.466 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:15.466 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:33:15.466 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:33:15.466 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:15.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.464+0000 7f2d74904640 1 librados: starting msgr at 2024-03-25T19:33:15.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.464+0000 7f2d74904640 1 librados: starting objecter 2024-03-25T19:33:15.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.464+0000 7f2d74904640 1 Processor -- start 2024-03-25T19:33:15.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.464+0000 7f2d74904640 1 -- start start 2024-03-25T19:33:15.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.464+0000 7f2d74904640 1 librados: setting wanted keys 2024-03-25T19:33:15.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.464+0000 7f2d74904640 1 librados: calling monclient init 2024-03-25T19:33:15.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.464+0000 7f2d74904640 10 monclient: init 2024-03-25T19:33:15.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.465+0000 7f2d74904640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:15.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.465+0000 7f2d74904640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:15.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.465+0000 7f2d74904640 10 monclient(hunting): picked mon.noname-c con 0x7f2d700a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:15.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.465+0000 7f2d74904640 10 monclient(hunting): picked mon.noname-b con 0x7f2d7013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:15.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.465+0000 7f2d74904640 10 monclient(hunting): picked mon.noname-a con 0x7f2d701430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:15.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.465+0000 7f2d74904640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d70138050 con 0x7f2d701430c0 2024-03-25T19:33:15.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.465+0000 7f2d74904640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d700abb70 con 0x7f2d7013f710 2024-03-25T19:33:15.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.465+0000 7f2d74904640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d701392c0 con 0x7f2d700a98f0 2024-03-25T19:33:15.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.465+0000 7f2d74904640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:15.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.465+0000 7f2d74904640 10 monclient(hunting): authenticate will time out at 6243.646827s 2024-03-25T19:33:15.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.477+0000 7f2d6e59c640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f2d700a98f0 0x7f2d70135480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:35516/0 (socket says 172.21.15.52:35516) 2024-03-25T19:33:15.477 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.477+0000 7f2d6e59c640 1 -- 172.21.15.52:0/2272719095 learned_addr learned my addr 172.21.15.52:0/2272719095 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:15.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.478+0000 7f2d677fe640 10 client.?.objecter ms_handle_connect 0x7f2d700a98f0 2024-03-25T19:33:15.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.478+0000 7f2d677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:15.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.479+0000 7f2d677fe640 10 client.?.objecter ms_handle_connect 0x7f2d701430c0 2024-03-25T19:33:15.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.479+0000 7f2d677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:15.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.489+0000 7f2d677fe640 10 client.?.objecter ms_handle_connect 0x7f2d7013f710 2024-03-25T19:33:15.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.489+0000 7f2d677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:15.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.495+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678089386 0 0) 0x7f2d701392c0 con 0x7f2d700a98f0 2024-03-25T19:33:15.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.495+0000 7f2d677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:15.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.495+0000 7f2d677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:15.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.495+0000 7f2d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:15.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.495+0000 7f2d677fe640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:33:15.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.495+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d50004eb0 con 0x7f2d700a98f0 2024-03-25T19:33:15.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.497+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149899694 0 0) 0x7f2d70138050 con 0x7f2d701430c0 2024-03-25T19:33:15.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.497+0000 7f2d677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:15.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.497+0000 7f2d677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:15.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.497+0000 7f2d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:15.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.497+0000 7f2d677fe640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:33:15.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.497+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d701392c0 con 0x7f2d701430c0 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.513+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2470883699 0 0) 0x7f2d701392c0 con 0x7f2d701430c0 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.513+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d70138050 con 0x7f2d701430c0 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.513+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2d68001a70 con 0x7f2d701430c0 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.513+0000 7f2d677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.513+0000 7f2d677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.513+0000 7f2d677fe640 10 monclient(hunting): dump: 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:15.514 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:15.515 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:15.515 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:15.515 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:15.515 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:15.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.513+0000 7f2d677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:15.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.515+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285472363 0 0) 0x7f2d700abb70 con 0x7f2d7013f710 2024-03-25T19:33:15.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.515+0000 7f2d677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:15.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.515+0000 7f2d677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:15.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.515+0000 7f2d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:15.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.515+0000 7f2d677fe640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:33:15.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.515+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d701392c0 con 0x7f2d7013f710 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.521+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4147910925 0 0) 0x7f2d50004eb0 con 0x7f2d700a98f0 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.521+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d700abb70 con 0x7f2d700a98f0 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.521+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2d600019b0 con 0x7f2d700a98f0 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.521+0000 7f2d677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.521+0000 7f2d677fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.521+0000 7f2d677fe640 10 monclient(hunting): dump: 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:15.522 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:15.523 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:15.523 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:15.523 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:15.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.534+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4210448492 0 0) 0x7f2d701392c0 con 0x7f2d7013f710 2024-03-25T19:33:15.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.534+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d50004eb0 con 0x7f2d7013f710 2024-03-25T19:33:15.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.534+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2d58001a20 con 0x7f2d7013f710 2024-03-25T19:33:15.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.534+0000 7f2d677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:15.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.534+0000 7f2d677fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:15.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.534+0000 7f2d677fe640 10 monclient(hunting): dump: 2024-03-25T19:33:15.535 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:15.535 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:15.535 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.535 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.535 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:15.535 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:15.535 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:15.535 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:15.535 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:15.535 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:15.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3493956174 0 0) 0x7f2d700abb70 con 0x7f2d700a98f0 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 >> v1:172.21.15.179:6789/0 conn(0x7f2d7013f710 legacy=0x7f2d701df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 >> v1:172.21.15.52:6789/0 conn(0x7f2d701430c0 legacy=0x7f2d701e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d677fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d677fe640 1 monclient: found mon.c 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d677fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d700abde0 con 0x7f2d700a98f0 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d677fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d677fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d74904640 5 monclient: authenticate success, global_id 4478 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d74904640 10 monclient: _renew_subs 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d74904640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d74904640 1 -- 172.21.15.52:0/2272719095 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d700ac0c0 con 0x7f2d700a98f0 2024-03-25T19:33:15.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d74904640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:15.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d74904640 10 monclient: _renew_subs 2024-03-25T19:33:15.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d74904640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:33:15.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d74904640 1 -- 172.21.15.52:0/2272719095 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7014e730 con 0x7f2d700a98f0 2024-03-25T19:33:15.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.536+0000 7f2d74904640 1 librados: init done 2024-03-25T19:33:15.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.537+0000 7f2d74904640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:15.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.537+0000 7f2d74904640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:15.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.537+0000 7f2d74904640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:33:15.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.537+0000 7f2d74904640 1 -- 172.21.15.52:0/2272719095 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d700627b0 con 0x7f2d700a98f0 2024-03-25T19:33:15.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.856+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d600047e0 con 0x7f2d700a98f0 2024-03-25T19:33:15.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.856+0000 7f2d677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:15.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.856+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2d60004af0 con 0x7f2d700a98f0 2024-03-25T19:33:15.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.856+0000 7f2d677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:15.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.856+0000 7f2d677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:15.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.856+0000 7f2d677fe640 10 monclient: dump: 2024-03-25T19:33:15.857 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:15.857 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:15.857 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.858 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:15.858 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:15.858 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:15.858 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:15.858 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:15.858 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:15.858 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:15.858 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.856+0000 7f2d6d59a640 4 set_mon_vals no callback set 2024-03-25T19:33:15.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.857+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f2d6001aac0 con 0x7f2d700a98f0 2024-03-25T19:33:15.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.858+0000 7f2d65d9b640 10 monclient: get_auth_request con 0x7f2d5006e110 auth_method 0 2024-03-25T19:33:15.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.858+0000 7f2d677fe640 10 client.4478.objecter ms_handle_connect 0x7f2d5006e110 2024-03-25T19:33:15.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.866+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f2d60054850 con 0x7f2d700a98f0 2024-03-25T19:33:15.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.867+0000 7f2d677fe640 10 client.4478.objecter ms_dispatch 0x7f2d70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:15.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.867+0000 7f2d677fe640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:15.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.867+0000 7f2d677fe640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:15.867 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.867+0000 7f2d677fe640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-25T19:33:15.868 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.867+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d60050190 con 0x7f2d700a98f0 2024-03-25T19:33:15.868 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.867+0000 7f2d677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:15.868 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:15.867+0000 7f2d677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:16.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.113+0000 7f2d74904640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:16.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.113+0000 7f2d74904640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:16.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.113+0000 7f2d74904640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:33:16.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.113+0000 7f2d74904640 1 -- 172.21.15.52:0/2272719095 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d7013de90 con 0x7f2d700a98f0 2024-03-25T19:33:16.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.147+0000 7f2d677fe640 1 -- 172.21.15.52:0/2272719095 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f2d60052290 con 0x7f2d700a98f0 2024-03-25T19:33:16.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.147+0000 7f2d677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:16.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.147+0000 7f2d677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:16.148 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:33:16.148 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:16.150 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.149+0000 7f2d74904640 10 librados: watch_flush enter 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.149+0000 7f2d74904640 10 librados: watch_flush exit 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.149+0000 7f2d74904640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.149+0000 7f2d74904640 10 client.4478.objecter successfully canceled tick 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.149+0000 7f2d74904640 1 -- 172.21.15.52:0/2272719095 >> v1:172.21.15.52:6800/3967427147 conn(0x7f2d5006e110 legacy=0x7f2d500705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.149+0000 7f2d74904640 10 monclient: shutdown 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.149+0000 7f2d74904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.149+0000 7f2d74904640 1 -- 172.21.15.52:0/2272719095 >> v1:172.21.15.52:6790/0 conn(0x7f2d700a98f0 legacy=0x7f2d70135480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.150+0000 7f2d74904640 1 -- 172.21.15.52:0/2272719095 shutdown_connections 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.150+0000 7f2d74904640 1 -- 172.21.15.52:0/2272719095 >> 172.21.15.52:0/2272719095 conn(0x7f2d700b7ea0 msgr2=0x7f2d70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.150+0000 7f2d74904640 1 -- 172.21.15.52:0/2272719095 shutdown_connections 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.150+0000 7f2d74904640 1 -- 172.21.15.52:0/2272719095 wait complete. 2024-03-25T19:33:16.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:16.150+0000 7f2d74904640 1 librados: shutdown 2024-03-25T19:33:16.162 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:22.970 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.086+0000 7ff74a6d9640 10 monclient: get_monmap_and_config 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.086+0000 7ff74a6d9640 10 monclient: build_initial_monmap 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.086+0000 7ff74a6d9640 10 monclient: monmap: 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:33:23.088332+0000 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:33:23.088332+0000 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:33:23.087 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:33:23.088 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:23.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.087+0000 7ff74a6d9640 1 Processor -- start 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.087+0000 7ff74a6d9640 1 -- start start 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.087+0000 7ff74a6d9640 10 monclient: init 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.088+0000 7ff74a6d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.088+0000 7ff74a6d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.088+0000 7ff74a6d9640 10 monclient(hunting): picked mon.noname-c con 0x7ff7440705b0 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.088+0000 7ff74a6d9640 10 monclient(hunting): picked mon.noname-b con 0x7ff7441402d0 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.088+0000 7ff74a6d9640 10 monclient(hunting): picked mon.noname-a con 0x7ff744149ae0 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.088+0000 7ff74a6d9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7440718c0 con 0x7ff744149ae0 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.088+0000 7ff74a6d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7440714c0 con 0x7ff7440705b0 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.088+0000 7ff74a6d9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff744072b50 con 0x7ff7441402d0 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.088+0000 7ff74a6d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:23.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.088+0000 7ff74a6d9640 10 monclient(hunting): authenticate will time out at 6251.269324s 2024-03-25T19:33:23.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.099+0000 7ff743fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7ff7440705b0 0x7ff74413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:46726/0 (socket says 172.21.15.52:46726) 2024-03-25T19:33:23.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.099+0000 7ff743fff640 1 -- 172.21.15.52:0/1289016741 learned_addr learned my addr 172.21.15.52:0/1289016741 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:23.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.121+0000 7ff742ffd640 1 -- 172.21.15.52:0/1289016741 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825229486 0 0) 0x7ff7440718c0 con 0x7ff744149ae0 2024-03-25T19:33:23.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.121+0000 7ff742ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:23.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.121+0000 7ff742ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:23.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.121+0000 7ff742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:23.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.121+0000 7ff742ffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:33:23.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.121+0000 7ff742ffd640 1 -- 172.21.15.52:0/1289016741 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff72c003520 con 0x7ff744149ae0 2024-03-25T19:33:23.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.122+0000 7ff742ffd640 1 -- 172.21.15.52:0/1289016741 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885859171 0 0) 0x7ff7440714c0 con 0x7ff7440705b0 2024-03-25T19:33:23.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.122+0000 7ff742ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:23.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.122+0000 7ff742ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:23.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.122+0000 7ff742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:23.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.122+0000 7ff742ffd640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:33:23.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.122+0000 7ff742ffd640 1 -- 172.21.15.52:0/1289016741 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7440718c0 con 0x7ff7440705b0 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 1 -- 172.21.15.52:0/1289016741 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2907095184 0 0) 0x7ff72c003520 con 0x7ff744149ae0 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 1 -- 172.21.15.52:0/1289016741 >> v1:172.21.15.52:6790/0 conn(0x7ff7441402d0 legacy=0x7ff744140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 1 -- 172.21.15.52:0/1289016741 >> v1:172.21.15.179:6789/0 conn(0x7ff7440705b0 legacy=0x7ff74413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 1 monclient: found mon.noname-a 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 1 -- 172.21.15.52:0/1289016741 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74413dd10 con 0x7ff744149ae0 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 10 monclient: _finish_auth 0 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 1 -- 172.21.15.52:0/1289016741 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff73c001a10 con 0x7ff744149ae0 2024-03-25T19:33:23.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff742ffd640 10 monclient: dump: 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff74a6d9640 5 monclient: authenticate success, global_id 4488 2024-03-25T19:33:23.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.128+0000 7ff74a6d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:23.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.141+0000 7ff742ffd640 1 -- 172.21.15.52:0/1289016741 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff73c001b70 con 0x7ff744149ae0 2024-03-25T19:33:23.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.141+0000 7ff742ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:23.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.141+0000 7ff742ffd640 1 -- 172.21.15.52:0/1289016741 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff73c001f80 con 0x7ff744149ae0 2024-03-25T19:33:23.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.141+0000 7ff742ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:23.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.141+0000 7ff742ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:23.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.141+0000 7ff742ffd640 10 monclient: dump: 2024-03-25T19:33:23.142 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:23.142 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:23.142 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.142 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.141+0000 7ff74a6d9640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:23.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.141+0000 7ff74a6d9640 4 set_mon_vals no callback set 2024-03-25T19:33:23.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.142+0000 7ff74a6d9640 10 monclient: shutdown 2024-03-25T19:33:23.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.142+0000 7ff74a6d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.142+0000 7ff74a6d9640 1 -- 172.21.15.52:0/1289016741 >> v1:172.21.15.52:6789/0 conn(0x7ff744149ae0 legacy=0x7ff74414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.142+0000 7ff74a6d9640 1 -- 172.21.15.52:0/1289016741 shutdown_connections 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.142+0000 7ff74a6d9640 1 -- 172.21.15.52:0/1289016741 >> 172.21.15.52:0/1289016741 conn(0x7ff7440b5d10 msgr2=0x7ff7440b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.142+0000 7ff74a6d9640 1 -- 172.21.15.52:0/1289016741 shutdown_connections 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.142+0000 7ff74a6d9640 1 -- 172.21.15.52:0/1289016741 wait complete. 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.143+0000 7ff74a6d9640 10 monclient: build_initial_monmap 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.143+0000 7ff74a6d9640 10 monclient: monmap: 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:33:23.145 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:33:23.146 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:23.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.143+0000 7ff74a6d9640 1 librados: starting msgr at 2024-03-25T19:33:23.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.143+0000 7ff74a6d9640 1 librados: starting objecter 2024-03-25T19:33:23.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.144+0000 7ff74a6d9640 1 Processor -- start 2024-03-25T19:33:23.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.144+0000 7ff74a6d9640 1 -- start start 2024-03-25T19:33:23.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.144+0000 7ff74a6d9640 1 librados: setting wanted keys 2024-03-25T19:33:23.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.144+0000 7ff74a6d9640 1 librados: calling monclient init 2024-03-25T19:33:23.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.144+0000 7ff74a6d9640 10 monclient: init 2024-03-25T19:33:23.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.145+0000 7ff74a6d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:23.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.145+0000 7ff74a6d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:23.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.145+0000 7ff74a6d9640 10 monclient(hunting): picked mon.noname-a con 0x7ff7440705b0 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:23.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.145+0000 7ff74a6d9640 10 monclient(hunting): picked mon.noname-b con 0x7ff7441402d0 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:23.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.145+0000 7ff74a6d9640 10 monclient(hunting): picked mon.noname-c con 0x7ff744149ae0 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:23.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.145+0000 7ff74a6d9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74406d800 con 0x7ff7440705b0 2024-03-25T19:33:23.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.145+0000 7ff74a6d9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74406dae0 con 0x7ff7441402d0 2024-03-25T19:33:23.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.145+0000 7ff74a6d9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74406c440 con 0x7ff744149ae0 2024-03-25T19:33:23.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.145+0000 7ff74a6d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:23.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.145+0000 7ff74a6d9640 10 monclient(hunting): authenticate will time out at 6251.326265s 2024-03-25T19:33:23.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.152+0000 7ff7437fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7ff7441402d0 0x7ff744134040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:46748/0 (socket says 172.21.15.52:46748) 2024-03-25T19:33:23.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.152+0000 7ff7437fe640 1 -- 172.21.15.52:0/830831477 learned_addr learned my addr 172.21.15.52:0/830831477 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:23.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.155+0000 7ff740ff9640 10 client.?.objecter ms_handle_connect 0x7ff7441402d0 2024-03-25T19:33:23.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.155+0000 7ff740ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:23.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.156+0000 7ff740ff9640 10 client.?.objecter ms_handle_connect 0x7ff744149ae0 2024-03-25T19:33:23.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.156+0000 7ff740ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:23.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.158+0000 7ff740ff9640 10 client.?.objecter ms_handle_connect 0x7ff7440705b0 2024-03-25T19:33:23.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.158+0000 7ff740ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:23.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.163+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259706692 0 0) 0x7ff74406dae0 con 0x7ff7441402d0 2024-03-25T19:33:23.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.163+0000 7ff740ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:23.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.163+0000 7ff740ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:23.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.163+0000 7ff740ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:23.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.163+0000 7ff740ff9640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:33:23.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.163+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff730004eb0 con 0x7ff7441402d0 2024-03-25T19:33:23.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.167+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384086010 0 0) 0x7ff74406c440 con 0x7ff744149ae0 2024-03-25T19:33:23.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.167+0000 7ff740ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:23.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.167+0000 7ff740ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:23.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.167+0000 7ff740ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:23.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.167+0000 7ff740ff9640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:33:23.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.167+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff74406dae0 con 0x7ff744149ae0 2024-03-25T19:33:23.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.170+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929672212 0 0) 0x7ff74406d800 con 0x7ff7440705b0 2024-03-25T19:33:23.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.170+0000 7ff740ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:23.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.170+0000 7ff740ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:23.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.170+0000 7ff740ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:23.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.170+0000 7ff740ff9640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:33:23.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.170+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff74406c440 con 0x7ff7440705b0 2024-03-25T19:33:23.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4250898826 0 0) 0x7ff730004eb0 con 0x7ff7441402d0 2024-03-25T19:33:23.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff74406d800 con 0x7ff7441402d0 2024-03-25T19:33:23.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff7380019d0 con 0x7ff7441402d0 2024-03-25T19:33:23.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:23.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:23.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:23.184 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:23.184 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1840183578 0 0) 0x7ff74406dae0 con 0x7ff744149ae0 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff730004eb0 con 0x7ff744149ae0 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff73c001e80 con 0x7ff744149ae0 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:23.185 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210850053 0 0) 0x7ff74406c440 con 0x7ff7440705b0 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff74406dae0 con 0x7ff7440705b0 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff734001930 con 0x7ff7440705b0 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:23.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.183+0000 7ff740ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:23.187 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:23.187 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:23.187 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.187 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.187 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:23.187 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:23.187 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:23.187 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:23.187 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:23.187 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:23.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1000437793 0 0) 0x7ff74406d800 con 0x7ff7441402d0 2024-03-25T19:33:23.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 >> v1:172.21.15.52:6790/0 conn(0x7ff744149ae0 legacy=0x7ff7441e2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 >> v1:172.21.15.52:6789/0 conn(0x7ff7440705b0 legacy=0x7ff744141910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff740ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:23.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff740ff9640 1 monclient: found mon.b 2024-03-25T19:33:23.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff740ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:23.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff740ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:23.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff744142020 con 0x7ff7441402d0 2024-03-25T19:33:23.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff740ff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:23.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff740ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:23.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff740ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:23.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff74a6d9640 5 monclient: authenticate success, global_id 4498 2024-03-25T19:33:23.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff74a6d9640 10 monclient: _renew_subs 2024-03-25T19:33:23.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff74a6d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:23.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff74a6d9640 1 -- 172.21.15.52:0/830831477 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff744142270 con 0x7ff7441402d0 2024-03-25T19:33:23.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff74a6d9640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:23.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff74a6d9640 10 monclient: _renew_subs 2024-03-25T19:33:23.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff74a6d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:23.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff74a6d9640 1 -- 172.21.15.52:0/830831477 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74406cb40 con 0x7ff7441402d0 2024-03-25T19:33:23.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.200+0000 7ff74a6d9640 1 librados: init done 2024-03-25T19:33:23.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.201+0000 7ff74a6d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:23.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.201+0000 7ff74a6d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:23.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.201+0000 7ff74a6d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:23.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.201+0000 7ff74a6d9640 1 -- 172.21.15.52:0/830831477 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74406c2a0 con 0x7ff7441402d0 2024-03-25T19:33:23.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.492+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff738003610 con 0x7ff7441402d0 2024-03-25T19:33:23.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.492+0000 7ff740ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:23.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.492+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff738003880 con 0x7ff7441402d0 2024-03-25T19:33:23.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.492+0000 7ff740ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:23.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.492+0000 7ff740ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:23.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.492+0000 7ff740ff9640 10 monclient: dump: 2024-03-25T19:33:23.493 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:23.493 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:23.493 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.493 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:23.494 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:23.494 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:23.494 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:23.494 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:23.494 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:23.494 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:23.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.492+0000 7ff7427fc640 4 set_mon_vals no callback set 2024-03-25T19:33:23.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.493+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7ff738003aa0 con 0x7ff7441402d0 2024-03-25T19:33:23.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.494+0000 7ff743fff640 10 monclient: get_auth_request con 0x7ff73006e0f0 auth_method 0 2024-03-25T19:33:23.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.494+0000 7ff740ff9640 10 client.4498.objecter ms_handle_connect 0x7ff73006e0f0 2024-03-25T19:33:23.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.496+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7ff738054690 con 0x7ff7441402d0 2024-03-25T19:33:23.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.496+0000 7ff740ff9640 10 client.4498.objecter ms_dispatch 0x7ff744000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:23.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.496+0000 7ff740ff9640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:23.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.496+0000 7ff740ff9640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:23.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.496+0000 7ff740ff9640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T19:33:23.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.500+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7380501a0 con 0x7ff7441402d0 2024-03-25T19:33:23.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.500+0000 7ff740ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:23.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.500+0000 7ff740ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:23.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.738+0000 7ff74a6d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:23.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.738+0000 7ff74a6d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:23.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.738+0000 7ff74a6d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:23.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.738+0000 7ff74a6d9640 1 -- 172.21.15.52:0/830831477 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff74406c5d0 con 0x7ff7441402d0 2024-03-25T19:33:23.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.769+0000 7ff740ff9640 1 -- 172.21.15.52:0/830831477 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7ff7380520d0 con 0x7ff7441402d0 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.769+0000 7ff740ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.769+0000 7ff740ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:33:23.770 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:23.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 10 librados: watch_flush enter 2024-03-25T19:33:23.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 10 librados: watch_flush exit 2024-03-25T19:33:23.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T19:33:23.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 10 client.4498.objecter successfully canceled tick 2024-03-25T19:33:23.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 1 -- 172.21.15.52:0/830831477 >> v1:172.21.15.52:6800/3967427147 conn(0x7ff73006e0f0 legacy=0x7ff7300705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 10 monclient: shutdown 2024-03-25T19:33:23.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:23.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 1 -- 172.21.15.52:0/830831477 >> v1:172.21.15.179:6789/0 conn(0x7ff7441402d0 legacy=0x7ff744134040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:23.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 1 -- 172.21.15.52:0/830831477 shutdown_connections 2024-03-25T19:33:23.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 1 -- 172.21.15.52:0/830831477 >> 172.21.15.52:0/830831477 conn(0x7ff7440b5d10 msgr2=0x7ff74413c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:23.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 1 -- 172.21.15.52:0/830831477 shutdown_connections 2024-03-25T19:33:23.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 1 -- 172.21.15.52:0/830831477 wait complete. 2024-03-25T19:33:23.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:23.771+0000 7ff74a6d9640 1 librados: shutdown 2024-03-25T19:33:23.785 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:30.593 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:30.707 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.706+0000 7fbdfc31c640 10 monclient: get_monmap_and_config 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.706+0000 7fbdfc31c640 10 monclient: build_initial_monmap 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.706+0000 7fbdfc31c640 10 monclient: monmap: 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:33:30.708496+0000 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:33:30.708496+0000 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:30.708 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.707+0000 7fbdfc31c640 1 Processor -- start 2024-03-25T19:33:30.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.707+0000 7fbdfc31c640 1 -- start start 2024-03-25T19:33:30.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.707+0000 7fbdfc31c640 10 monclient: init 2024-03-25T19:33:30.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.708+0000 7fbdfc31c640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:30.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.708+0000 7fbdfc31c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:30.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.708+0000 7fbdfc31c640 10 monclient(hunting): picked mon.noname-a con 0x7fbdf40a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:30.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.708+0000 7fbdfc31c640 10 monclient(hunting): picked mon.noname-c con 0x7fbdf413f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:30.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.708+0000 7fbdfc31c640 10 monclient(hunting): picked mon.noname-b con 0x7fbdf41430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:30.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.708+0000 7fbdfc31c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf4142520 con 0x7fbdf40a98f0 2024-03-25T19:33:30.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.708+0000 7fbdfc31c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf40b28a0 con 0x7fbdf413f710 2024-03-25T19:33:30.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.708+0000 7fbdfc31c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf40b2a30 con 0x7fbdf41430c0 2024-03-25T19:33:30.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.708+0000 7fbdfc31c640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:30.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.708+0000 7fbdfc31c640 10 monclient(hunting): authenticate will time out at 6258.889692s 2024-03-25T19:33:30.722 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.721+0000 7fbdfa8b9640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fbdf41430c0 0x7fbdf414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:55342/0 (socket says 172.21.15.52:55342) 2024-03-25T19:33:30.722 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.721+0000 7fbdfa8b9640 1 -- 172.21.15.52:0/3347194593 learned_addr learned my addr 172.21.15.52:0/3347194593 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:30.733 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.733+0000 7fbdf90b6640 1 -- 172.21.15.52:0/3347194593 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179616327 0 0) 0x7fbdf40b28a0 con 0x7fbdf413f710 2024-03-25T19:33:30.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.733+0000 7fbdf90b6640 10 monclient(hunting): _negotiate 2024-03-25T19:33:30.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.733+0000 7fbdf90b6640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:30.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.733+0000 7fbdf90b6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:30.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.733+0000 7fbdf90b6640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:33:30.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.733+0000 7fbdf90b6640 1 -- 172.21.15.52:0/3347194593 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde0003520 con 0x7fbdf413f710 2024-03-25T19:33:30.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 1 -- 172.21.15.52:0/3347194593 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4253268215 0 0) 0x7fbde0003520 con 0x7fbdf413f710 2024-03-25T19:33:30.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 1 -- 172.21.15.52:0/3347194593 >> v1:172.21.15.52:6790/0 conn(0x7fbdf41430c0 legacy=0x7fbdf414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 1 -- 172.21.15.52:0/3347194593 >> v1:172.21.15.52:6789/0 conn(0x7fbdf40a98f0 legacy=0x7fbdf40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 10 monclient: _finish_hunting 0 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 1 monclient: found mon.noname-c 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 1 -- 172.21.15.52:0/3347194593 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf40b44c0 con 0x7fbdf413f710 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 10 monclient: _finish_auth 0 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 10 monclient: _check_auth_tickets 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 1 -- 172.21.15.52:0/3347194593 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fbde4001690 con 0x7fbdf413f710 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdf90b6640 10 monclient: dump: 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:30.746 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:30.747 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:30.747 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:30.747 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:30.747 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:30.747 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:30.747 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:30.747 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:30.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdfc31c640 5 monclient: authenticate success, global_id 4501 2024-03-25T19:33:30.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.744+0000 7fbdfc31c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:30.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.755+0000 7fbdf90b6640 1 -- 172.21.15.52:0/3347194593 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbde40017f0 con 0x7fbdf413f710 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.755+0000 7fbdf90b6640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.755+0000 7fbdf90b6640 1 -- 172.21.15.52:0/3347194593 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fbde4001c00 con 0x7fbdf413f710 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.755+0000 7fbdf90b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.755+0000 7fbdf90b6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.755+0000 7fbdf90b6640 10 monclient: dump: 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:30.756 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:30.757 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:30.757 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:30.757 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:30.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.755+0000 7fbdfc31c640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:30.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.755+0000 7fbdfc31c640 4 set_mon_vals no callback set 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.756+0000 7fbdfc31c640 10 monclient: shutdown 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.756+0000 7fbdfc31c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.756+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3347194593 >> v1:172.21.15.179:6789/0 conn(0x7fbdf413f710 legacy=0x7fbdf41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.756+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3347194593 shutdown_connections 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.756+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3347194593 >> 172.21.15.52:0/3347194593 conn(0x7fbdf40b7ea0 msgr2=0x7fbdf40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.756+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3347194593 shutdown_connections 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.756+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3347194593 wait complete. 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.757+0000 7fbdfc31c640 10 monclient: build_initial_monmap 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.757+0000 7fbdfc31c640 10 monclient: monmap: 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:30.759 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.757+0000 7fbdfc31c640 1 librados: starting msgr at 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.757+0000 7fbdfc31c640 1 librados: starting objecter 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.757+0000 7fbdfc31c640 1 Processor -- start 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.757+0000 7fbdfc31c640 1 -- start start 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.757+0000 7fbdfc31c640 1 librados: setting wanted keys 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.757+0000 7fbdfc31c640 1 librados: calling monclient init 2024-03-25T19:33:30.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.757+0000 7fbdfc31c640 10 monclient: init 2024-03-25T19:33:30.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.758+0000 7fbdfc31c640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:30.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.758+0000 7fbdfc31c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:30.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.758+0000 7fbdfc31c640 10 monclient(hunting): picked mon.noname-c con 0x7fbdf40a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:30.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.758+0000 7fbdfc31c640 10 monclient(hunting): picked mon.noname-b con 0x7fbdf413f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:30.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.758+0000 7fbdfc31c640 10 monclient(hunting): picked mon.noname-a con 0x7fbdf41430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:30.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.758+0000 7fbdfc31c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf40abdc0 con 0x7fbdf41430c0 2024-03-25T19:33:30.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.758+0000 7fbdfc31c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf41d7730 con 0x7fbdf413f710 2024-03-25T19:33:30.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.758+0000 7fbdfc31c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf4138f10 con 0x7fbdf40a98f0 2024-03-25T19:33:30.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.758+0000 7fbdfc31c640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:30.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.758+0000 7fbdfc31c640 10 monclient(hunting): authenticate will time out at 6258.939517s 2024-03-25T19:33:30.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.763+0000 7fbdfa8b9640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fbdf41430c0 0x7fbdf41d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:46752/0 (socket says 172.21.15.52:46752) 2024-03-25T19:33:30.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.763+0000 7fbdfa8b9640 1 -- 172.21.15.52:0/3921249834 learned_addr learned my addr 172.21.15.52:0/3921249834 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:30.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.768+0000 7fbdeaffd640 10 client.?.objecter ms_handle_connect 0x7fbdf41430c0 2024-03-25T19:33:30.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.768+0000 7fbdeaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:30.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.771+0000 7fbdeaffd640 10 client.?.objecter ms_handle_connect 0x7fbdf40a98f0 2024-03-25T19:33:30.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.771+0000 7fbdeaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:30.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.782+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578658400 0 0) 0x7fbdf4138f10 con 0x7fbdf40a98f0 2024-03-25T19:33:30.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.782+0000 7fbdeaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:30.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.782+0000 7fbdeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:30.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.782+0000 7fbdeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:30.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.782+0000 7fbdeaffd640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:33:30.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.783+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8004eb0 con 0x7fbdf40a98f0 2024-03-25T19:33:30.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.787+0000 7fbdeaffd640 10 client.?.objecter ms_handle_connect 0x7fbdf413f710 2024-03-25T19:33:30.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.787+0000 7fbdeaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:30.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.788+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713248690 0 0) 0x7fbdf40abdc0 con 0x7fbdf41430c0 2024-03-25T19:33:30.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.788+0000 7fbdeaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:30.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.788+0000 7fbdeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:30.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.788+0000 7fbdeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:30.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.788+0000 7fbdeaffd640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:33:30.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.788+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf4138f10 con 0x7fbdf41430c0 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.800+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3163984086 0 0) 0x7fbdd8004eb0 con 0x7fbdf40a98f0 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.800+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdf40abdc0 con 0x7fbdf40a98f0 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.800+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fbdf0001930 con 0x7fbdf40a98f0 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.800+0000 7fbdeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.800+0000 7fbdeaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.800+0000 7fbdeaffd640 10 monclient(hunting): dump: 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:30.801 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:30.802 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:30.802 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:30.802 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:30.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.800+0000 7fbdeaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:30.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.801+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849035769 0 0) 0x7fbdf41d7730 con 0x7fbdf413f710 2024-03-25T19:33:30.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.801+0000 7fbdeaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:30.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.801+0000 7fbdeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:30.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.801+0000 7fbdeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:30.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.801+0000 7fbdeaffd640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:33:30.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.801+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8004eb0 con 0x7fbdf413f710 2024-03-25T19:33:30.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.819+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3057059484 0 0) 0x7fbdf4138f10 con 0x7fbdf41430c0 2024-03-25T19:33:30.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.819+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdf41d7730 con 0x7fbdf41430c0 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.819+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fbdec001b50 con 0x7fbdf41430c0 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.819+0000 7fbdeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.819+0000 7fbdeaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.819+0000 7fbdeaffd640 10 monclient(hunting): dump: 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:30.821 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 559640915 0 0) 0x7fbdf40abdc0 con 0x7fbdf40a98f0 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 >> v1:172.21.15.179:6789/0 conn(0x7fbdf413f710 legacy=0x7fbdf41d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 >> v1:172.21.15.52:6789/0 conn(0x7fbdf41430c0 legacy=0x7fbdf41d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdeaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdeaffd640 1 monclient: found mon.c 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdeaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf41390a0 con 0x7fbdf40a98f0 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdeaffd640 10 monclient: _finish_auth 0 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdeaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdfc31c640 5 monclient: authenticate success, global_id 4490 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdfc31c640 10 monclient: _renew_subs 2024-03-25T19:33:30.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdfc31c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:33:30.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3921249834 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf41d8320 con 0x7fbdf40a98f0 2024-03-25T19:33:30.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdfc31c640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:30.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdfc31c640 10 monclient: _renew_subs 2024-03-25T19:33:30.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdfc31c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:33:30.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3921249834 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf41d43c0 con 0x7fbdf40a98f0 2024-03-25T19:33:30.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.820+0000 7fbdfc31c640 1 librados: init done 2024-03-25T19:33:30.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.821+0000 7fbdfc31c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:30.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.821+0000 7fbdfc31c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:30.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.821+0000 7fbdfc31c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:33:30.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:30.821+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3921249834 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf40627b0 con 0x7fbdf40a98f0 2024-03-25T19:33:31.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.154+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdf00035f0 con 0x7fbdf40a98f0 2024-03-25T19:33:31.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.154+0000 7fbdeaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:31.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.154+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fbdf0003980 con 0x7fbdf40a98f0 2024-03-25T19:33:31.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.154+0000 7fbdeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:31.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.154+0000 7fbdeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:31.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.154+0000 7fbdeaffd640 10 monclient: dump: 2024-03-25T19:33:31.155 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:31.156 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:31.156 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:31.156 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:31.156 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:31.156 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:31.156 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:31.156 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:31.156 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:31.156 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:31.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.154+0000 7fbdf88b5640 4 set_mon_vals no callback set 2024-03-25T19:33:31.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.155+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fbdf0003ba0 con 0x7fbdf40a98f0 2024-03-25T19:33:31.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.156+0000 7fbdf98b7640 10 monclient: get_auth_request con 0x7fbdd806dfb0 auth_method 0 2024-03-25T19:33:31.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.156+0000 7fbdeaffd640 10 client.4490.objecter ms_handle_connect 0x7fbdd806dfb0 2024-03-25T19:33:31.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.162+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fbdf0053590 con 0x7fbdf40a98f0 2024-03-25T19:33:31.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.162+0000 7fbdeaffd640 10 client.4490.objecter ms_dispatch 0x7fbdf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:31.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.162+0000 7fbdeaffd640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:31.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.162+0000 7fbdeaffd640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:31.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.162+0000 7fbdeaffd640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-25T19:33:31.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.162+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbdf004f0a0 con 0x7fbdf40a98f0 2024-03-25T19:33:31.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.162+0000 7fbdeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:31.163 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.162+0000 7fbdeaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:31.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.396+0000 7fbdfc31c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:31.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.396+0000 7fbdfc31c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:31.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.396+0000 7fbdfc31c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:33:31.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.396+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3921249834 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdf4069f40 con 0x7fbdf40a98f0 2024-03-25T19:33:31.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.436+0000 7fbdeaffd640 1 -- 172.21.15.52:0/3921249834 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fbdf0050fd0 con 0x7fbdf40a98f0 2024-03-25T19:33:31.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.436+0000 7fbdeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:31.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.436+0000 7fbdeaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:31.437 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:33:31.437 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:31.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.438+0000 7fbdfc31c640 10 librados: watch_flush enter 2024-03-25T19:33:31.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.438+0000 7fbdfc31c640 10 librados: watch_flush exit 2024-03-25T19:33:31.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.438+0000 7fbdfc31c640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-25T19:33:31.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.438+0000 7fbdfc31c640 10 client.4490.objecter successfully canceled tick 2024-03-25T19:33:31.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.438+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3921249834 >> v1:172.21.15.52:6800/3967427147 conn(0x7fbdd806dfb0 legacy=0x7fbdd8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:31.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.439+0000 7fbdfc31c640 10 monclient: shutdown 2024-03-25T19:33:31.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.439+0000 7fbdfc31c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:31.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.439+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3921249834 >> v1:172.21.15.52:6790/0 conn(0x7fbdf40a98f0 legacy=0x7fbdf4138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:31.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.439+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3921249834 shutdown_connections 2024-03-25T19:33:31.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.439+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3921249834 >> 172.21.15.52:0/3921249834 conn(0x7fbdf40b7ea0 msgr2=0x7fbdf4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:31.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.439+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3921249834 shutdown_connections 2024-03-25T19:33:31.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.439+0000 7fbdfc31c640 1 -- 172.21.15.52:0/3921249834 wait complete. 2024-03-25T19:33:31.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:31.439+0000 7fbdfc31c640 1 librados: shutdown 2024-03-25T19:33:31.452 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:32.120 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:06:25.718 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.129 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:06:25.728 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.174 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:06:25.773 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.186 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:06:25.728 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.196 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:06:25.795 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.199 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:06:25.741 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.200 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:06:25.799 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.222 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:06:25.764 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:32.235 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:06:25.833 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:38.258 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:38.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.371+0000 7f855f4ed640 10 monclient: get_monmap_and_config 2024-03-25T19:33:38.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.371+0000 7f855f4ed640 10 monclient: build_initial_monmap 2024-03-25T19:33:38.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.371+0000 7f855f4ed640 10 monclient: monmap: 2024-03-25T19:33:38.372 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:38.372 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:38.372 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:33:38.372588+0000 2024-03-25T19:33:38.372 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:33:38.372588+0000 2024-03-25T19:33:38.372 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:38.372 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:38.372 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:38.372 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:33:38.372 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:33:38.372 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:38.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.371+0000 7f855f4ed640 1 Processor -- start 2024-03-25T19:33:38.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.371+0000 7f855f4ed640 1 -- start start 2024-03-25T19:33:38.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.371+0000 7f855f4ed640 10 monclient: init 2024-03-25T19:33:38.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.372+0000 7f855f4ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:38.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.372+0000 7f855f4ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:38.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.372+0000 7f855f4ed640 10 monclient(hunting): picked mon.noname-b con 0x7f85580a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:38.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.372+0000 7f855f4ed640 10 monclient(hunting): picked mon.noname-a con 0x7f855813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:38.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.372+0000 7f855f4ed640 10 monclient(hunting): picked mon.noname-c con 0x7f85581430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:38.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.372+0000 7f855f4ed640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8558142520 con 0x7f855813f710 2024-03-25T19:33:38.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.372+0000 7f855f4ed640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85580b28a0 con 0x7f85581430c0 2024-03-25T19:33:38.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.372+0000 7f855f4ed640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85580b2a30 con 0x7f85580a98f0 2024-03-25T19:33:38.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.372+0000 7f855f4ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:38.375 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.372+0000 7f855f4ed640 10 monclient(hunting): authenticate will time out at 6266.553505s 2024-03-25T19:33:38.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.376+0000 7f855ca88640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f855813f710 0x7f85581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:32844/0 (socket says 172.21.15.52:32844) 2024-03-25T19:33:38.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.376+0000 7f855ca88640 1 -- 172.21.15.52:0/2203079674 learned_addr learned my addr 172.21.15.52:0/2203079674 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:38.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.386+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593007745 0 0) 0x7f8558142520 con 0x7f855813f710 2024-03-25T19:33:38.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.386+0000 7f854ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:38.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.386+0000 7f854ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:38.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.386+0000 7f854ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:38.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.386+0000 7f854ffff640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:33:38.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.386+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8544003520 con 0x7f855813f710 2024-03-25T19:33:38.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.394+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465063040 0 0) 0x7f85580b2a30 con 0x7f85580a98f0 2024-03-25T19:33:38.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.394+0000 7f854ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:38.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.394+0000 7f854ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:38.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.394+0000 7f854ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:38.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.394+0000 7f854ffff640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:33:38.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.394+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8558142520 con 0x7f85580a98f0 2024-03-25T19:33:38.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916087048 0 0) 0x7f85580b28a0 con 0x7f85581430c0 2024-03-25T19:33:38.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:38.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:38.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:38.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:33:38.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85580b2a30 con 0x7f85581430c0 2024-03-25T19:33:38.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3592496728 0 0) 0x7f8544003520 con 0x7f855813f710 2024-03-25T19:33:38.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 >> v1:172.21.15.52:6790/0 conn(0x7f85580a98f0 legacy=0x7f85580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:38.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 >> v1:172.21.15.179:6789/0 conn(0x7f85581430c0 legacy=0x7f855814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:38.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:33:38.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 1 monclient: found mon.noname-a 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85580b44c0 con 0x7f855813f710 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 10 monclient: _finish_auth 0 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8548001690 con 0x7f855813f710 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f854ffff640 10 monclient: dump: 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:38.406 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:38.407 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:38.407 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:38.407 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:38.407 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:38.407 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:38.407 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:38.407 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:38.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f855f4ed640 5 monclient: authenticate success, global_id 4500 2024-03-25T19:33:38.407 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.404+0000 7f855f4ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:38.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.420+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85480017f0 con 0x7f855813f710 2024-03-25T19:33:38.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.420+0000 7f854ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:38.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.420+0000 7f854ffff640 1 -- 172.21.15.52:0/2203079674 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8548001c00 con 0x7f855813f710 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.420+0000 7f854ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.420+0000 7f854ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.420+0000 7f854ffff640 10 monclient: dump: 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:38.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.420+0000 7f855f4ed640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:38.422 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.420+0000 7f855f4ed640 4 set_mon_vals no callback set 2024-03-25T19:33:38.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.421+0000 7f855f4ed640 10 monclient: shutdown 2024-03-25T19:33:38.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.421+0000 7f855f4ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:38.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.421+0000 7f855f4ed640 1 -- 172.21.15.52:0/2203079674 >> v1:172.21.15.52:6789/0 conn(0x7f855813f710 legacy=0x7f85581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:38.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.421+0000 7f855f4ed640 1 -- 172.21.15.52:0/2203079674 shutdown_connections 2024-03-25T19:33:38.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.421+0000 7f855f4ed640 1 -- 172.21.15.52:0/2203079674 >> 172.21.15.52:0/2203079674 conn(0x7f85580b7ea0 msgr2=0x7f85580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:38.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.421+0000 7f855f4ed640 1 -- 172.21.15.52:0/2203079674 shutdown_connections 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.421+0000 7f855f4ed640 1 -- 172.21.15.52:0/2203079674 wait complete. 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.422+0000 7f855f4ed640 10 monclient: build_initial_monmap 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.422+0000 7f855f4ed640 10 monclient: monmap: 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.422+0000 7f855f4ed640 1 librados: starting msgr at 2024-03-25T19:33:38.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.422+0000 7f855f4ed640 1 librados: starting objecter 2024-03-25T19:33:38.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.422+0000 7f855f4ed640 1 Processor -- start 2024-03-25T19:33:38.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.422+0000 7f855f4ed640 1 -- start start 2024-03-25T19:33:38.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.422+0000 7f855f4ed640 1 librados: setting wanted keys 2024-03-25T19:33:38.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.422+0000 7f855f4ed640 1 librados: calling monclient init 2024-03-25T19:33:38.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.422+0000 7f855f4ed640 10 monclient: init 2024-03-25T19:33:38.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.423+0000 7f855f4ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:38.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.423+0000 7f855f4ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:38.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.423+0000 7f855f4ed640 10 monclient(hunting): picked mon.noname-a con 0x7f85580a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:38.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.423+0000 7f855f4ed640 10 monclient(hunting): picked mon.noname-c con 0x7f855813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:38.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.423+0000 7f855f4ed640 10 monclient(hunting): picked mon.noname-b con 0x7f85581430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:38.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.423+0000 7f855f4ed640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8558137080 con 0x7f85580a98f0 2024-03-25T19:33:38.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.423+0000 7f855f4ed640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8558137360 con 0x7f85581430c0 2024-03-25T19:33:38.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.424+0000 7f855f4ed640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8558135b10 con 0x7f855813f710 2024-03-25T19:33:38.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.424+0000 7f855f4ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:38.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.424+0000 7f855f4ed640 10 monclient(hunting): authenticate will time out at 6266.604967s 2024-03-25T19:33:38.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.432+0000 7f855d289640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f85580a98f0 0x7f8558137a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:32882/0 (socket says 172.21.15.52:32882) 2024-03-25T19:33:38.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.432+0000 7f855d289640 1 -- 172.21.15.52:0/774096631 learned_addr learned my addr 172.21.15.52:0/774096631 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:38.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.433+0000 7f854dffb640 10 client.?.objecter ms_handle_connect 0x7f85580a98f0 2024-03-25T19:33:38.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.433+0000 7f854dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:38.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.441+0000 7f854dffb640 10 client.?.objecter ms_handle_connect 0x7f85581430c0 2024-03-25T19:33:38.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.441+0000 7f854dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:38.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.442+0000 7f854dffb640 10 client.?.objecter ms_handle_connect 0x7f855813f710 2024-03-25T19:33:38.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.442+0000 7f854dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:38.443 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.443+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298124359 0 0) 0x7f8558137080 con 0x7f85580a98f0 2024-03-25T19:33:38.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.443+0000 7f854dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:38.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.443+0000 7f854dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:38.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.443+0000 7f854dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:38.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.443+0000 7f854dffb640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:33:38.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.443+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f853c004eb0 con 0x7f85580a98f0 2024-03-25T19:33:38.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.448+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484879860 0 0) 0x7f8558135b10 con 0x7f855813f710 2024-03-25T19:33:38.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.448+0000 7f854dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:38.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.448+0000 7f854dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:38.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.448+0000 7f854dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:38.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.448+0000 7f854dffb640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:33:38.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.448+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8558137080 con 0x7f855813f710 2024-03-25T19:33:38.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.450+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790070139 0 0) 0x7f8558137360 con 0x7f85581430c0 2024-03-25T19:33:38.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.450+0000 7f854dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:33:38.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.450+0000 7f854dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:38.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.450+0000 7f854dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:38.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.450+0000 7f854dffb640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:33:38.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.450+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8558135b10 con 0x7f85581430c0 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.451+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 347260391 0 0) 0x7f853c004eb0 con 0x7f85580a98f0 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.451+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8558137360 con 0x7f85580a98f0 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.451+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8554001a30 con 0x7f85580a98f0 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.451+0000 7f854dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.451+0000 7f854dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.451+0000 7f854dffb640 10 monclient(hunting): dump: 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:38.452 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:38.453 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:38.453 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:38.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.451+0000 7f854dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:38.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.464+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2295238169 0 0) 0x7f8558137080 con 0x7f855813f710 2024-03-25T19:33:38.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f853c004eb0 con 0x7f855813f710 2024-03-25T19:33:38.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8548001b00 con 0x7f855813f710 2024-03-25T19:33:38.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:38.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:38.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 10 monclient(hunting): dump: 2024-03-25T19:33:38.466 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:38.466 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:38.466 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:38.466 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:38.466 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:38.466 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:38.466 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:38.466 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:38.466 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:38.466 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:38.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2823022057 0 0) 0x7f8558137360 con 0x7f85580a98f0 2024-03-25T19:33:38.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 >> v1:172.21.15.52:6790/0 conn(0x7f855813f710 legacy=0x7f85580b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:38.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 >> v1:172.21.15.179:6789/0 conn(0x7f85581430c0 legacy=0x7f8558130340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:38.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:33:38.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 1 monclient: found mon.a 2024-03-25T19:33:38.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:38.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:33:38.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8558135df0 con 0x7f85580a98f0 2024-03-25T19:33:38.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 10 monclient: _finish_auth 0 2024-03-25T19:33:38.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:33:38.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f854dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f855f4ed640 5 monclient: authenticate success, global_id 4503 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f855f4ed640 10 monclient: _renew_subs 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f855f4ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f855f4ed640 1 -- 172.21.15.52:0/774096631 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8558136340 con 0x7f85580a98f0 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f855f4ed640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f855f4ed640 10 monclient: _renew_subs 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f855f4ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f855f4ed640 1 -- 172.21.15.52:0/774096631 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85581352e0 con 0x7f85580a98f0 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.465+0000 7f855f4ed640 1 librados: init done 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.467+0000 7f855f4ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.467+0000 7f855f4ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.467+0000 7f855f4ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:33:38.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.467+0000 7f855f4ed640 1 -- 172.21.15.52:0/774096631 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85580627b0 con 0x7f85580a98f0 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.786+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8554001c70 con 0x7f85580a98f0 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.786+0000 7f854dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.786+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8554003a60 con 0x7f85580a98f0 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.786+0000 7f854dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.786+0000 7f854dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.786+0000 7f854dffb640 10 monclient: dump: 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:38.787 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:38.788 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:38.788 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:38.788 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:38.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.786+0000 7f854f7fe640 4 set_mon_vals no callback set 2024-03-25T19:33:38.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.787+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f8554003bc0 con 0x7f85580a98f0 2024-03-25T19:33:38.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.788+0000 7f855ca88640 10 monclient: get_auth_request con 0x7f853c06df70 auth_method 0 2024-03-25T19:33:38.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.788+0000 7f854dffb640 10 client.4503.objecter ms_handle_connect 0x7f853c06df70 2024-03-25T19:33:38.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.794+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f8554054940 con 0x7f85580a98f0 2024-03-25T19:33:38.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.794+0000 7f854dffb640 10 client.4503.objecter ms_dispatch 0x7f8558000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:38.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.794+0000 7f854dffb640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:38.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.794+0000 7f854dffb640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:38.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.794+0000 7f854dffb640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-25T19:33:38.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.794+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8554050450 con 0x7f85580a98f0 2024-03-25T19:33:38.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.795+0000 7f854dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:38.795 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:38.795+0000 7f854dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:39.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.009+0000 7f855f4ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:39.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.009+0000 7f855f4ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:39.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.009+0000 7f855f4ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:33:39.010 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.009+0000 7f855f4ed640 1 -- 172.21.15.52:0/774096631 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8558074cc0 con 0x7f85580a98f0 2024-03-25T19:33:39.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.046+0000 7f854dffb640 1 -- 172.21.15.52:0/774096631 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f8554052380 con 0x7f85580a98f0 2024-03-25T19:33:39.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.046+0000 7f854dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:39.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.046+0000 7f854dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:39.047 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:33:39.047 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:39.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 10 librados: watch_flush enter 2024-03-25T19:33:39.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 10 librados: watch_flush exit 2024-03-25T19:33:39.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-25T19:33:39.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 10 client.4503.objecter successfully canceled tick 2024-03-25T19:33:39.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 1 -- 172.21.15.52:0/774096631 >> v1:172.21.15.52:6800/3967427147 conn(0x7f853c06df70 legacy=0x7f853c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:39.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 10 monclient: shutdown 2024-03-25T19:33:39.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:39.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 1 -- 172.21.15.52:0/774096631 >> v1:172.21.15.52:6789/0 conn(0x7f85580a98f0 legacy=0x7f8558137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:39.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 1 -- 172.21.15.52:0/774096631 shutdown_connections 2024-03-25T19:33:39.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 1 -- 172.21.15.52:0/774096631 >> 172.21.15.52:0/774096631 conn(0x7f85580b7ea0 msgr2=0x7f8558074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:39.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 1 -- 172.21.15.52:0/774096631 shutdown_connections 2024-03-25T19:33:39.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 1 -- 172.21.15.52:0/774096631 wait complete. 2024-03-25T19:33:39.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:39.049+0000 7f855f4ed640 1 librados: shutdown 2024-03-25T19:33:39.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:45.866 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:45.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.973+0000 7f31b6d82640 10 monclient: get_monmap_and_config 2024-03-25T19:33:45.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.973+0000 7f31b6d82640 10 monclient: build_initial_monmap 2024-03-25T19:33:45.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.973+0000 7f31b6d82640 10 monclient: monmap: 2024-03-25T19:33:45.974 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:45.974 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:45.974 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:33:45.975316+0000 2024-03-25T19:33:45.974 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:33:45.975316+0000 2024-03-25T19:33:45.975 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:45.975 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:45.975 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:45.975 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:33:45.975 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:33:45.975 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.974+0000 7f31b6d82640 1 Processor -- start 2024-03-25T19:33:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.974+0000 7f31b6d82640 1 -- start start 2024-03-25T19:33:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.974+0000 7f31b6d82640 10 monclient: init 2024-03-25T19:33:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.975+0000 7f31b6d82640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:45.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.975+0000 7f31b6d82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:45.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.975+0000 7f31b6d82640 10 monclient(hunting): picked mon.noname-a con 0x7f31b00b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:45.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.975+0000 7f31b6d82640 10 monclient(hunting): picked mon.noname-b con 0x7f31b00b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:45.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.975+0000 7f31b6d82640 10 monclient(hunting): picked mon.noname-c con 0x7f31b014be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:45.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.975+0000 7f31b6d82640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b00af940 con 0x7f31b00b4d60 2024-03-25T19:33:45.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.975+0000 7f31b6d82640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b00b0a70 con 0x7f31b014be90 2024-03-25T19:33:45.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.975+0000 7f31b6d82640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b00b0d50 con 0x7f31b00b1730 2024-03-25T19:33:45.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.975+0000 7f31b6d82640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:45.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.975+0000 7f31b6d82640 10 monclient(hunting): authenticate will time out at 6274.156082s 2024-03-25T19:33:45.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.991+0000 7f31b531f640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f31b014be90 0x7f31b014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:60566/0 (socket says 172.21.15.52:60566) 2024-03-25T19:33:45.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:45.991+0000 7f31b531f640 1 -- 172.21.15.52:0/260472320 learned_addr learned my addr 172.21.15.52:0/260472320 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:46.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.005+0000 7f31af7fe640 1 -- 172.21.15.52:0/260472320 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086693752 0 0) 0x7f31b00b0a70 con 0x7f31b014be90 2024-03-25T19:33:46.006 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.005+0000 7f31af7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:46.006 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.005+0000 7f31af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:46.006 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.005+0000 7f31af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:46.006 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.005+0000 7f31af7fe640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:33:46.006 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.005+0000 7f31af7fe640 1 -- 172.21.15.52:0/260472320 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a0003520 con 0x7f31b014be90 2024-03-25T19:33:46.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 1 -- 172.21.15.52:0/260472320 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3263867180 0 0) 0x7f31a0003520 con 0x7f31b014be90 2024-03-25T19:33:46.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 1 -- 172.21.15.52:0/260472320 >> v1:172.21.15.52:6790/0 conn(0x7f31b00b1730 legacy=0x7f31b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 1 -- 172.21.15.52:0/260472320 >> v1:172.21.15.52:6789/0 conn(0x7f31b00b4d60 legacy=0x7f31b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:46.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 1 monclient: found mon.noname-c 2024-03-25T19:33:46.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:46.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:33:46.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 1 -- 172.21.15.52:0/260472320 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b00b1030 con 0x7f31b014be90 2024-03-25T19:33:46.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:46.018 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 1 -- 172.21.15.52:0/260472320 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f31a8001a10 con 0x7f31b014be90 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31b6d82640 5 monclient: authenticate success, global_id 4513 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31b6d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31af7fe640 10 monclient: dump: 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:46.019 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:46.020 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:46.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.017+0000 7f31b6d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:46.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.028+0000 7f31af7fe640 1 -- 172.21.15.52:0/260472320 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31a8001b70 con 0x7f31b014be90 2024-03-25T19:33:46.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.028+0000 7f31af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:46.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.028+0000 7f31af7fe640 1 -- 172.21.15.52:0/260472320 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f31a8001f80 con 0x7f31b014be90 2024-03-25T19:33:46.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.028+0000 7f31af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:46.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.028+0000 7f31af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:46.029 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.028+0000 7f31af7fe640 10 monclient: dump: 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.028+0000 7f31b6d82640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:46.030 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.028+0000 7f31b6d82640 4 set_mon_vals no callback set 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.029+0000 7f31b6d82640 10 monclient: shutdown 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.029+0000 7f31b6d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.029+0000 7f31b6d82640 1 -- 172.21.15.52:0/260472320 >> v1:172.21.15.179:6789/0 conn(0x7f31b014be90 legacy=0x7f31b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.029+0000 7f31b6d82640 1 -- 172.21.15.52:0/260472320 shutdown_connections 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.029+0000 7f31b6d82640 1 -- 172.21.15.52:0/260472320 >> 172.21.15.52:0/260472320 conn(0x7f31b00617a0 msgr2=0x7f31b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.029+0000 7f31b6d82640 1 -- 172.21.15.52:0/260472320 shutdown_connections 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.029+0000 7f31b6d82640 1 -- 172.21.15.52:0/260472320 wait complete. 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.030+0000 7f31b6d82640 10 monclient: build_initial_monmap 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.030+0000 7f31b6d82640 10 monclient: monmap: 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:33:46.032 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.031+0000 7f31b6d82640 1 librados: starting msgr at 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.031+0000 7f31b6d82640 1 librados: starting objecter 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.031+0000 7f31b6d82640 1 Processor -- start 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.031+0000 7f31b6d82640 1 -- start start 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.031+0000 7f31b6d82640 1 librados: setting wanted keys 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.031+0000 7f31b6d82640 1 librados: calling monclient init 2024-03-25T19:33:46.033 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.031+0000 7f31b6d82640 10 monclient: init 2024-03-25T19:33:46.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.032+0000 7f31b6d82640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:46.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.032+0000 7f31b6d82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:46.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.032+0000 7f31b6d82640 10 monclient(hunting): picked mon.noname-a con 0x7f31b00b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:46.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.032+0000 7f31b6d82640 10 monclient(hunting): picked mon.noname-b con 0x7f31b00b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:46.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.032+0000 7f31b6d82640 10 monclient(hunting): picked mon.noname-c con 0x7f31b014be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:46.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.032+0000 7f31b6d82640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b0140c10 con 0x7f31b00b1730 2024-03-25T19:33:46.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.032+0000 7f31b6d82640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b0134900 con 0x7f31b00b4d60 2024-03-25T19:33:46.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.032+0000 7f31b6d82640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b0141a70 con 0x7f31b014be90 2024-03-25T19:33:46.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.032+0000 7f31b6d82640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:46.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.032+0000 7f31b6d82640 10 monclient(hunting): authenticate will time out at 6274.213521s 2024-03-25T19:33:46.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.042+0000 7f31affff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f31b00b4d60 0x7f31b0133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:60590/0 (socket says 172.21.15.52:60590) 2024-03-25T19:33:46.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.042+0000 7f31affff640 1 -- 172.21.15.52:0/1547121472 learned_addr learned my addr 172.21.15.52:0/1547121472 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:46.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.043+0000 7f31ad7fa640 10 client.?.objecter ms_handle_connect 0x7f31b014be90 2024-03-25T19:33:46.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.043+0000 7f31ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:46.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.043+0000 7f31ad7fa640 10 client.?.objecter ms_handle_connect 0x7f31b00b4d60 2024-03-25T19:33:46.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.043+0000 7f31ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:46.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.047+0000 7f31ad7fa640 10 client.?.objecter ms_handle_connect 0x7f31b00b1730 2024-03-25T19:33:46.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.047+0000 7f31ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:46.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.049+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965427882 0 0) 0x7f31b0134900 con 0x7f31b00b4d60 2024-03-25T19:33:46.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.049+0000 7f31ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:46.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.049+0000 7f31ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:46.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.049+0000 7f31ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:46.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.049+0000 7f31ad7fa640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:33:46.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.049+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3194004eb0 con 0x7f31b00b4d60 2024-03-25T19:33:46.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.055+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441054933 0 0) 0x7f31b0140c10 con 0x7f31b00b1730 2024-03-25T19:33:46.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.055+0000 7f31ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:46.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.055+0000 7f31ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:46.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.055+0000 7f31ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:46.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.055+0000 7f31ad7fa640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:33:46.056 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.055+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b0134900 con 0x7f31b00b1730 2024-03-25T19:33:46.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.058+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980520234 0 0) 0x7f31b0141a70 con 0x7f31b014be90 2024-03-25T19:33:46.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.058+0000 7f31ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:46.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.058+0000 7f31ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:46.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.058+0000 7f31ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:46.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.058+0000 7f31ad7fa640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:33:46.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.058+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b0140c10 con 0x7f31b014be90 2024-03-25T19:33:46.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.069+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1974098183 0 0) 0x7f3194004eb0 con 0x7f31b00b4d60 2024-03-25T19:33:46.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.070+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31b0141a70 con 0x7f31b00b4d60 2024-03-25T19:33:46.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.070+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f319c001a30 con 0x7f31b00b4d60 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.070+0000 7f31ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.070+0000 7f31ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.070+0000 7f31ad7fa640 10 monclient(hunting): dump: 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:46.071 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:46.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.070+0000 7f31ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.074+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1431210026 0 0) 0x7f31b0140c10 con 0x7f31b014be90 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.074+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3194004eb0 con 0x7f31b014be90 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.074+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f31a8001e80 con 0x7f31b014be90 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.074+0000 7f31ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.074+0000 7f31ad7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.074+0000 7f31ad7fa640 10 monclient(hunting): dump: 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:46.075 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:46.076 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:46.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.079+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 701760420 0 0) 0x7f31b0134900 con 0x7f31b00b1730 2024-03-25T19:33:46.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.079+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31b0140c10 con 0x7f31b00b1730 2024-03-25T19:33:46.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.079+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f31a40018a0 con 0x7f31b00b1730 2024-03-25T19:33:46.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.079+0000 7f31ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:46.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.079+0000 7f31ad7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:46.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.079+0000 7f31ad7fa640 10 monclient(hunting): dump: 2024-03-25T19:33:46.081 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:46.081 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:46.081 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.081 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.081 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:46.081 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:46.081 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:46.081 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:46.081 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:46.081 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:46.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1810615301 0 0) 0x7f31b0141a70 con 0x7f31b00b4d60 2024-03-25T19:33:46.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 >> v1:172.21.15.52:6790/0 conn(0x7f31b014be90 legacy=0x7f31b01341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 >> v1:172.21.15.52:6789/0 conn(0x7f31b00b1730 legacy=0x7f31b0142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31ad7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:33:46.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31ad7fa640 1 monclient: found mon.b 2024-03-25T19:33:46.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31ad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0141d50 con 0x7f31b00b4d60 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31ad7fa640 10 monclient: _finish_auth 0 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31ad7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31b6d82640 5 monclient: authenticate success, global_id 4516 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31b6d82640 10 monclient: _renew_subs 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31b6d82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31b6d82640 1 -- 172.21.15.52:0/1547121472 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b0140380 con 0x7f31b00b4d60 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31b6d82640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31b6d82640 10 monclient: _renew_subs 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31b6d82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:46.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31b6d82640 1 -- 172.21.15.52:0/1547121472 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b01389e0 con 0x7f31b00b4d60 2024-03-25T19:33:46.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.090+0000 7f31b6d82640 1 librados: init done 2024-03-25T19:33:46.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.092+0000 7f31b6d82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:46.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.092+0000 7f31b6d82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:46.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.092+0000 7f31b6d82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:46.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.092+0000 7f31b6d82640 1 -- 172.21.15.52:0/1547121472 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b0134b60 con 0x7f31b00b4d60 2024-03-25T19:33:46.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.355+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f319c001c70 con 0x7f31b00b4d60 2024-03-25T19:33:46.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.355+0000 7f31ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.355+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f319c003a60 con 0x7f31b00b4d60 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.355+0000 7f31ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.355+0000 7f31ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.355+0000 7f31ad7fa640 10 monclient: dump: 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:46.357 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:46.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.355+0000 7f31aeffd640 4 set_mon_vals no callback set 2024-03-25T19:33:46.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.356+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f319c003c40 con 0x7f31b00b4d60 2024-03-25T19:33:46.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.357+0000 7f31b4b1e640 10 monclient: get_auth_request con 0x7f319406e0f0 auth_method 0 2024-03-25T19:33:46.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.357+0000 7f31ad7fa640 10 client.4516.objecter ms_handle_connect 0x7f319406e0f0 2024-03-25T19:33:46.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.359+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f319c054890 con 0x7f31b00b4d60 2024-03-25T19:33:46.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.359+0000 7f31ad7fa640 10 client.4516.objecter ms_dispatch 0x7f31b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:46.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.359+0000 7f31ad7fa640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:46.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.359+0000 7f31ad7fa640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:46.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.359+0000 7f31ad7fa640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T19:33:46.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.362+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f319c0503a0 con 0x7f31b00b4d60 2024-03-25T19:33:46.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.362+0000 7f31ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:46.362 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.362+0000 7f31ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:46.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.618+0000 7f31b6d82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:46.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.618+0000 7f31b6d82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:46.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.618+0000 7f31b6d82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:46.619 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.618+0000 7f31b6d82640 1 -- 172.21.15.52:0/1547121472 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31b0140930 con 0x7f31b00b4d60 2024-03-25T19:33:46.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.797+0000 7f31ad7fa640 1 -- 172.21.15.52:0/1547121472 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f31b0140930 con 0x7f31b00b4d60 2024-03-25T19:33:46.799 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.797+0000 7f31ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:46.799 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.797+0000 7f31ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:46.799 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:33:46.799 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.799+0000 7f31b6d82640 10 librados: watch_flush enter 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.799+0000 7f31b6d82640 10 librados: watch_flush exit 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.799+0000 7f31b6d82640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.799+0000 7f31b6d82640 10 client.4516.objecter successfully canceled tick 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.799+0000 7f31b6d82640 1 -- 172.21.15.52:0/1547121472 >> v1:172.21.15.52:6800/3967427147 conn(0x7f319406e0f0 legacy=0x7f31940705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.799+0000 7f31b6d82640 10 monclient: shutdown 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.799+0000 7f31b6d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.799+0000 7f31b6d82640 1 -- 172.21.15.52:0/1547121472 >> v1:172.21.15.179:6789/0 conn(0x7f31b00b4d60 legacy=0x7f31b0133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.800+0000 7f31b6d82640 1 -- 172.21.15.52:0/1547121472 shutdown_connections 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.800+0000 7f31b6d82640 1 -- 172.21.15.52:0/1547121472 >> 172.21.15.52:0/1547121472 conn(0x7f31b00617a0 msgr2=0x7f31b013b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.800+0000 7f31b6d82640 1 -- 172.21.15.52:0/1547121472 shutdown_connections 2024-03-25T19:33:46.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.800+0000 7f31b6d82640 1 -- 172.21.15.52:0/1547121472 wait complete. 2024-03-25T19:33:46.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:46.800+0000 7f31b6d82640 1 librados: shutdown 2024-03-25T19:33:46.812 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:52.213 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:06:45.811 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.221 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:06:45.820 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.241 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:06:45.840 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.285 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:06:45.827 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.287 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:06:45.886 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.291 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:06:45.833 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.294 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:06:45.893 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.310 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:06:45.852 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:52.318 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:06:45.916 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:53.618 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:53.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.735+0000 7f8a74e93640 10 monclient: get_monmap_and_config 2024-03-25T19:33:53.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.735+0000 7f8a74e93640 10 monclient: build_initial_monmap 2024-03-25T19:33:53.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.735+0000 7f8a74e93640 10 monclient: monmap: 2024-03-25T19:33:53.735 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:53.736 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:53.736 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:33:53.736448+0000 2024-03-25T19:33:53.736 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:33:53.736448+0000 2024-03-25T19:33:53.736 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:53.736 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:53.736 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:53.736 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:33:53.736 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:33:53.736 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:53.737 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.736+0000 7f8a74e93640 1 Processor -- start 2024-03-25T19:33:53.737 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.736+0000 7f8a74e93640 1 -- start start 2024-03-25T19:33:53.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.736+0000 7f8a74e93640 10 monclient: init 2024-03-25T19:33:53.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.737+0000 7f8a74e93640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:53.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.737+0000 7f8a74e93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:53.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.737+0000 7f8a74e93640 10 monclient(hunting): picked mon.noname-a con 0x7f8a700a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:53.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.737+0000 7f8a74e93640 10 monclient(hunting): picked mon.noname-c con 0x7f8a7013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:53.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.737+0000 7f8a74e93640 10 monclient(hunting): picked mon.noname-b con 0x7f8a701430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:53.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.737+0000 7f8a74e93640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a70142520 con 0x7f8a700a98f0 2024-03-25T19:33:53.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.737+0000 7f8a74e93640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a700b28a0 con 0x7f8a7013f710 2024-03-25T19:33:53.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.737+0000 7f8a74e93640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a700b2a30 con 0x7f8a701430c0 2024-03-25T19:33:53.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.737+0000 7f8a74e93640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:53.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.737+0000 7f8a74e93640 10 monclient(hunting): authenticate will time out at 6281.917592s 2024-03-25T19:33:53.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.745+0000 7f8a67fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f8a7013f710 0x7f8a701429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:37012/0 (socket says 172.21.15.52:37012) 2024-03-25T19:33:53.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.745+0000 7f8a67fff640 1 -- 172.21.15.52:0/71500602 learned_addr learned my addr 172.21.15.52:0/71500602 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:53.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.762+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977090833 0 0) 0x7f8a700b28a0 con 0x7f8a7013f710 2024-03-25T19:33:53.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.762+0000 7f8a6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.762+0000 7f8a6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.762+0000 7f8a6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.762+0000 7f8a6dd9b640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:33:53.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.762+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a5c003520 con 0x7f8a7013f710 2024-03-25T19:33:53.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.763+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482174125 0 0) 0x7f8a70142520 con 0x7f8a700a98f0 2024-03-25T19:33:53.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.763+0000 7f8a6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.763+0000 7f8a6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.763+0000 7f8a6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.763+0000 7f8a6dd9b640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:33:53.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.763+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a700b28a0 con 0x7f8a700a98f0 2024-03-25T19:33:53.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.767+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727822055 0 0) 0x7f8a700b2a30 con 0x7f8a701430c0 2024-03-25T19:33:53.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.767+0000 7f8a6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.767+0000 7f8a6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.767+0000 7f8a6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.767+0000 7f8a6dd9b640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:33:53.767 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.767+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a70142520 con 0x7f8a701430c0 2024-03-25T19:33:53.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2896268210 0 0) 0x7f8a700b28a0 con 0x7f8a700a98f0 2024-03-25T19:33:53.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 >> v1:172.21.15.52:6790/0 conn(0x7f8a701430c0 legacy=0x7f8a7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 >> v1:172.21.15.179:6789/0 conn(0x7f8a7013f710 legacy=0x7f8a701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:33:53.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:33:53.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:53.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:33:53.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a700b44c0 con 0x7f8a700a98f0 2024-03-25T19:33:53.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8a60001670 con 0x7f8a700a98f0 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a6dd9b640 10 monclient: dump: 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:53.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a74e93640 5 monclient: authenticate success, global_id 4512 2024-03-25T19:33:53.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.771+0000 7f8a74e93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:53.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.778+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a600017d0 con 0x7f8a700a98f0 2024-03-25T19:33:53.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.778+0000 7f8a6dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:53.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.778+0000 7f8a6dd9b640 1 -- 172.21.15.52:0/71500602 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8a60001be0 con 0x7f8a700a98f0 2024-03-25T19:33:53.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.778+0000 7f8a6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.778+0000 7f8a6dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.778+0000 7f8a6dd9b640 10 monclient: dump: 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:53.778 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:53.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.778+0000 7f8a74e93640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:53.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.778+0000 7f8a74e93640 4 set_mon_vals no callback set 2024-03-25T19:33:53.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.778+0000 7f8a74e93640 10 monclient: shutdown 2024-03-25T19:33:53.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.778+0000 7f8a74e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:53.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.778+0000 7f8a74e93640 1 -- 172.21.15.52:0/71500602 >> v1:172.21.15.52:6789/0 conn(0x7f8a700a98f0 legacy=0x7f8a700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.779+0000 7f8a74e93640 1 -- 172.21.15.52:0/71500602 shutdown_connections 2024-03-25T19:33:53.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.779+0000 7f8a74e93640 1 -- 172.21.15.52:0/71500602 >> 172.21.15.52:0/71500602 conn(0x7f8a700b7ea0 msgr2=0x7f8a700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:53.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.779+0000 7f8a74e93640 1 -- 172.21.15.52:0/71500602 shutdown_connections 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.779+0000 7f8a74e93640 1 -- 172.21.15.52:0/71500602 wait complete. 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.780+0000 7f8a74e93640 10 monclient: build_initial_monmap 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.780+0000 7f8a74e93640 10 monclient: monmap: 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:53.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.780+0000 7f8a74e93640 1 librados: starting msgr at 2024-03-25T19:33:53.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.780+0000 7f8a74e93640 1 librados: starting objecter 2024-03-25T19:33:53.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.780+0000 7f8a74e93640 1 Processor -- start 2024-03-25T19:33:53.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.780+0000 7f8a74e93640 1 -- start start 2024-03-25T19:33:53.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.780+0000 7f8a74e93640 1 librados: setting wanted keys 2024-03-25T19:33:53.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.780+0000 7f8a74e93640 1 librados: calling monclient init 2024-03-25T19:33:53.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.780+0000 7f8a74e93640 10 monclient: init 2024-03-25T19:33:53.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.781+0000 7f8a74e93640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:53.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.781+0000 7f8a74e93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:53.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.781+0000 7f8a74e93640 10 monclient(hunting): picked mon.noname-b con 0x7f8a700a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:33:53.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.781+0000 7f8a74e93640 10 monclient(hunting): picked mon.noname-a con 0x7f8a7013f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:33:53.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.781+0000 7f8a74e93640 10 monclient(hunting): picked mon.noname-c con 0x7f8a701430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:33:53.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.781+0000 7f8a74e93640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a70138f10 con 0x7f8a7013f710 2024-03-25T19:33:53.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.781+0000 7f8a74e93640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a70139970 con 0x7f8a700a98f0 2024-03-25T19:33:53.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.781+0000 7f8a74e93640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a700abb50 con 0x7f8a701430c0 2024-03-25T19:33:53.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.781+0000 7f8a74e93640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:53.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.781+0000 7f8a74e93640 10 monclient(hunting): authenticate will time out at 6281.961552s 2024-03-25T19:33:53.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.786+0000 7f8a6e59c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f8a700a98f0 0x7f8a70136330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:37030/0 (socket says 172.21.15.52:37030) 2024-03-25T19:33:53.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.786+0000 7f8a6e59c640 1 -- 172.21.15.52:0/4080515710 learned_addr learned my addr 172.21.15.52:0/4080515710 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:33:53.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.787+0000 7f8a66ffd640 10 client.?.objecter ms_handle_connect 0x7f8a700a98f0 2024-03-25T19:33:53.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.787+0000 7f8a66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:53.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.796+0000 7f8a66ffd640 10 client.?.objecter ms_handle_connect 0x7f8a701430c0 2024-03-25T19:33:53.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.796+0000 7f8a66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:53.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.798+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083262224 0 0) 0x7f8a70139970 con 0x7f8a700a98f0 2024-03-25T19:33:53.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.798+0000 7f8a66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.798+0000 7f8a66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.798+0000 7f8a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.798+0000 7f8a66ffd640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:33:53.798 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.798+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a50004eb0 con 0x7f8a700a98f0 2024-03-25T19:33:53.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.801+0000 7f8a66ffd640 10 client.?.objecter ms_handle_connect 0x7f8a7013f710 2024-03-25T19:33:53.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.801+0000 7f8a66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:53.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.809+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576387071 0 0) 0x7f8a700abb50 con 0x7f8a701430c0 2024-03-25T19:33:53.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.809+0000 7f8a66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.809+0000 7f8a66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.809+0000 7f8a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.809+0000 7f8a66ffd640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:33:53.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.809+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a70139970 con 0x7f8a701430c0 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601967894 0 0) 0x7f8a70138f10 con 0x7f8a7013f710 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a700abb50 con 0x7f8a7013f710 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 310443825 0 0) 0x7f8a50004eb0 con 0x7f8a700a98f0 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a70138f10 con 0x7f8a700a98f0 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8a60001670 con 0x7f8a700a98f0 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 10 monclient(hunting): dump: 2024-03-25T19:33:53.812 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:53.813 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:53.813 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:53.813 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:53.813 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:53.813 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:53.813 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:53.813 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:53.813 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:53.813 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:53.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.811+0000 7f8a66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:53.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 886866934 0 0) 0x7f8a70138f10 con 0x7f8a700a98f0 2024-03-25T19:33:53.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 >> v1:172.21.15.52:6790/0 conn(0x7f8a701430c0 legacy=0x7f8a70138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 >> v1:172.21.15.52:6789/0 conn(0x7f8a7013f710 legacy=0x7f8a700ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a66ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:53.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a66ffd640 1 monclient: found mon.b 2024-03-25T19:33:53.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:53.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a66ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:53.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a700abe30 con 0x7f8a700a98f0 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a66ffd640 10 monclient: _finish_auth 0 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a66ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a74e93640 5 monclient: authenticate success, global_id 4522 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a74e93640 10 monclient: _renew_subs 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a74e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a74e93640 1 -- 172.21.15.52:0/4080515710 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a70137b40 con 0x7f8a700a98f0 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a74e93640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a74e93640 10 monclient: _renew_subs 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a74e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a74e93640 1 -- 172.21.15.52:0/4080515710 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a70138c10 con 0x7f8a700a98f0 2024-03-25T19:33:53.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.817+0000 7f8a74e93640 1 librados: init done 2024-03-25T19:33:53.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.818+0000 7f8a74e93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:53.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.818+0000 7f8a74e93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:53.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.818+0000 7f8a74e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:53.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.818+0000 7f8a74e93640 1 -- 172.21.15.52:0/4080515710 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a700627b0 con 0x7f8a700a98f0 2024-03-25T19:33:53.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.841+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a600038a0 con 0x7f8a700a98f0 2024-03-25T19:33:53.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.841+0000 7f8a66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.841+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f8a60003cf0 con 0x7f8a700a98f0 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.841+0000 7f8a66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.841+0000 7f8a66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.841+0000 7f8a66ffd640 10 monclient: dump: 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:33:53.841 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:33:53.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.841+0000 7f8a6d59a640 4 set_mon_vals no callback set 2024-03-25T19:33:53.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.842+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f8a60003f10 con 0x7f8a700a98f0 2024-03-25T19:33:53.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.842+0000 7f8a67fff640 10 monclient: get_auth_request con 0x7f8a5006ddf0 auth_method 0 2024-03-25T19:33:53.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.842+0000 7f8a66ffd640 10 client.4522.objecter ms_handle_connect 0x7f8a5006ddf0 2024-03-25T19:33:53.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.844+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f8a60054960 con 0x7f8a700a98f0 2024-03-25T19:33:53.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.844+0000 7f8a66ffd640 10 client.4522.objecter ms_dispatch 0x7f8a70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:53.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.844+0000 7f8a66ffd640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:53.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.844+0000 7f8a66ffd640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:53.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:53.844+0000 7f8a66ffd640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T19:33:54.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.117+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a60050470 con 0x7f8a700a98f0 2024-03-25T19:33:54.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.117+0000 7f8a66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:54.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.117+0000 7f8a66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:54.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.376+0000 7f8a74e93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:54.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.376+0000 7f8a74e93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:54.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.376+0000 7f8a74e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:33:54.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.376+0000 7f8a74e93640 1 -- 172.21.15.52:0/4080515710 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a7013db40 con 0x7f8a700a98f0 2024-03-25T19:33:54.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.399+0000 7f8a66ffd640 1 -- 172.21.15.52:0/4080515710 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f8a600523a0 con 0x7f8a700a98f0 2024-03-25T19:33:54.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.399+0000 7f8a66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:54.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.399+0000 7f8a66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:54.399 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:33:54.399 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.401+0000 7f8a74e93640 10 librados: watch_flush enter 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.401+0000 7f8a74e93640 10 librados: watch_flush exit 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.401+0000 7f8a74e93640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.401+0000 7f8a74e93640 10 client.4522.objecter successfully canceled tick 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.401+0000 7f8a74e93640 1 -- 172.21.15.52:0/4080515710 >> v1:172.21.15.52:6800/3967427147 conn(0x7f8a5006ddf0 legacy=0x7f8a500702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.401+0000 7f8a74e93640 10 monclient: shutdown 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.401+0000 7f8a74e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.401+0000 7f8a74e93640 1 -- 172.21.15.52:0/4080515710 >> v1:172.21.15.179:6789/0 conn(0x7f8a700a98f0 legacy=0x7f8a70136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.402+0000 7f8a74e93640 1 -- 172.21.15.52:0/4080515710 shutdown_connections 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.402+0000 7f8a74e93640 1 -- 172.21.15.52:0/4080515710 >> 172.21.15.52:0/4080515710 conn(0x7f8a700b7ea0 msgr2=0x7f8a70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.402+0000 7f8a74e93640 1 -- 172.21.15.52:0/4080515710 shutdown_connections 2024-03-25T19:33:54.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.402+0000 7f8a74e93640 1 -- 172.21.15.52:0/4080515710 wait complete. 2024-03-25T19:33:54.403 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:33:54.402+0000 7f8a74e93640 1 librados: shutdown 2024-03-25T19:33:54.414 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:01.221 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:01.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.331+0000 7fe4bfa0d640 10 monclient: get_monmap_and_config 2024-03-25T19:34:01.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.331+0000 7fe4bfa0d640 10 monclient: build_initial_monmap 2024-03-25T19:34:01.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.331+0000 7fe4bfa0d640 10 monclient: monmap: 2024-03-25T19:34:01.331 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:01.331 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:01.331 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:34:01.331936+0000 2024-03-25T19:34:01.331 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:34:01.331936+0000 2024-03-25T19:34:01.331 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:01.332 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:01.332 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:01.332 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:34:01.332 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:34:01.332 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:01.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 1 Processor -- start 2024-03-25T19:34:01.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 1 -- start start 2024-03-25T19:34:01.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 10 monclient: init 2024-03-25T19:34:01.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:01.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:01.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b80a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:01.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:01.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b81430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:01.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8142520 con 0x7fe4b81430c0 2024-03-25T19:34:01.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80b28a0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80b2a30 con 0x7fe4b813f710 2024-03-25T19:34:01.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:01.334 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.332+0000 7fe4bfa0d640 10 monclient(hunting): authenticate will time out at 6289.512815s 2024-03-25T19:34:01.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.337+0000 7fe4bcfa8640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fe4b813f710 0x7fe4b81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:47612/0 (socket says 172.21.15.52:47612) 2024-03-25T19:34:01.336 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.337+0000 7fe4bcfa8640 1 -- 172.21.15.52:0/2523268200 learned_addr learned my addr 172.21.15.52:0/2523268200 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:01.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.352+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921152506 0 0) 0x7fe4b80b28a0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.352+0000 7fe4affff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:01.352 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.352+0000 7fe4affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.352+0000 7fe4affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.352+0000 7fe4affff640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:34:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.352+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a4003520 con 0x7fe4b80a98f0 2024-03-25T19:34:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.353+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563403012 0 0) 0x7fe4b80b2a30 con 0x7fe4b813f710 2024-03-25T19:34:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.353+0000 7fe4affff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.353+0000 7fe4affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.353+0000 7fe4affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.353+0000 7fe4affff640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:34:01.353 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.353+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b80b28a0 con 0x7fe4b813f710 2024-03-25T19:34:01.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.357+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362344837 0 0) 0x7fe4b8142520 con 0x7fe4b81430c0 2024-03-25T19:34:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.357+0000 7fe4affff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.357+0000 7fe4affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.357+0000 7fe4affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.357+0000 7fe4affff640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:34:01.358 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.357+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b80b2a30 con 0x7fe4b81430c0 2024-03-25T19:34:01.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1201983489 0 0) 0x7fe4a4003520 con 0x7fe4b80a98f0 2024-03-25T19:34:01.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 >> v1:172.21.15.52:6790/0 conn(0x7fe4b813f710 legacy=0x7fe4b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:01.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 >> v1:172.21.15.52:6789/0 conn(0x7fe4b81430c0 legacy=0x7fe4b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:01.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:01.364 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 1 monclient: found mon.noname-c 2024-03-25T19:34:01.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:01.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:34:01.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b80b44c0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 10 monclient: _finish_auth 0 2024-03-25T19:34:01.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:01.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:01.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fe4b4001690 con 0x7fe4b80a98f0 2024-03-25T19:34:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4bfa0d640 5 monclient: authenticate success, global_id 4525 2024-03-25T19:34:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4bfa0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:01.366 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4affff640 10 monclient: dump: 2024-03-25T19:34:01.366 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:01.367 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:01.367 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:01.367 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:01.367 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:01.367 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:01.367 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:01.367 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:01.367 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:01.368 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:01.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.363+0000 7fe4bfa0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:01.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.366+0000 7fe4affff640 1 -- 172.21.15.52:0/2523268200 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b40017f0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.371 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.366+0000 7fe4affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:01.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.366+0000 7fe4bfa0d640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:01.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.366+0000 7fe4bfa0d640 4 set_mon_vals no callback set 2024-03-25T19:34:01.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.367+0000 7fe4bfa0d640 10 monclient: shutdown 2024-03-25T19:34:01.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.367+0000 7fe4bfa0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:01.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.367+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/2523268200 >> v1:172.21.15.179:6789/0 conn(0x7fe4b80a98f0 legacy=0x7fe4b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:01.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.367+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/2523268200 shutdown_connections 2024-03-25T19:34:01.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.367+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/2523268200 >> 172.21.15.52:0/2523268200 conn(0x7fe4b80b7ea0 msgr2=0x7fe4b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:01.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.367+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/2523268200 shutdown_connections 2024-03-25T19:34:01.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.367+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/2523268200 wait complete. 2024-03-25T19:34:01.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.368+0000 7fe4bfa0d640 10 monclient: build_initial_monmap 2024-03-25T19:34:01.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.368+0000 7fe4bfa0d640 10 monclient: monmap: 2024-03-25T19:34:01.374 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:01.374 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:01.374 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:34:01.374 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:34:01.375 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:01.375 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:34:01.375 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:01.375 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:34:01.375 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:34:01.375 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:01.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.368+0000 7fe4bfa0d640 1 librados: starting msgr at 2024-03-25T19:34:01.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.368+0000 7fe4bfa0d640 1 librados: starting objecter 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.368+0000 7fe4bfa0d640 1 Processor -- start 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.368+0000 7fe4bfa0d640 1 -- start start 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.368+0000 7fe4bfa0d640 1 librados: setting wanted keys 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.368+0000 7fe4bfa0d640 1 librados: calling monclient init 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.368+0000 7fe4bfa0d640 10 monclient: init 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.369+0000 7fe4bfa0d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.369+0000 7fe4bfa0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.369+0000 7fe4bfa0d640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b80a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.369+0000 7fe4bfa0d640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.369+0000 7fe4bfa0d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b81430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.369+0000 7fe4bfa0d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80ab9e0 con 0x7fe4b813f710 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.369+0000 7fe4bfa0d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8138920 con 0x7fe4b81430c0 2024-03-25T19:34:01.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.369+0000 7fe4bfa0d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8138ab0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.369+0000 7fe4bfa0d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:01.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.369+0000 7fe4bfa0d640 10 monclient(hunting): authenticate will time out at 6289.549432s 2024-03-25T19:34:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.381+0000 7fe4bd7a9640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fe4b80a98f0 0x7fe4b8138210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:47636/0 (socket says 172.21.15.52:47636) 2024-03-25T19:34:01.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.381+0000 7fe4bd7a9640 1 -- 172.21.15.52:0/3676169040 learned_addr learned my addr 172.21.15.52:0/3676169040 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:01.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.382+0000 7fe4adffb640 10 client.?.objecter ms_handle_connect 0x7fe4b80a98f0 2024-03-25T19:34:01.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.382+0000 7fe4adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:01.381 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.382+0000 7fe4adffb640 10 client.?.objecter ms_handle_connect 0x7fe4b813f710 2024-03-25T19:34:01.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.382+0000 7fe4adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:01.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.389+0000 7fe4adffb640 10 client.?.objecter ms_handle_connect 0x7fe4b81430c0 2024-03-25T19:34:01.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.389+0000 7fe4adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:01.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.390+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370280713 0 0) 0x7fe4b8138ab0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.390+0000 7fe4adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:01.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.390+0000 7fe4adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:01.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.390+0000 7fe4adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:01.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.390+0000 7fe4adffb640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:34:01.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.390+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c004eb0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.397+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247161398 0 0) 0x7fe4b8138920 con 0x7fe4b81430c0 2024-03-25T19:34:01.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.397+0000 7fe4adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:01.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.397+0000 7fe4adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:01.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.397+0000 7fe4adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:01.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.397+0000 7fe4adffb640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:34:01.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.397+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b8138ab0 con 0x7fe4b81430c0 2024-03-25T19:34:01.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.402+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272646653 0 0) 0x7fe4b80ab9e0 con 0x7fe4b813f710 2024-03-25T19:34:01.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.402+0000 7fe4adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:01.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.402+0000 7fe4adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:01.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.402+0000 7fe4adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:01.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.402+0000 7fe4adffb640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:34:01.402 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.402+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b8138920 con 0x7fe4b813f710 2024-03-25T19:34:01.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.413+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 590310538 0 0) 0x7fe49c004eb0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.413+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b80ab9e0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.413+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fe4b4001d60 con 0x7fe4b80a98f0 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.413+0000 7fe4adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.413+0000 7fe4adffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.413+0000 7fe4adffb640 10 monclient(hunting): dump: 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:01.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.413+0000 7fe4adffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:01.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.426+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2029637054 0 0) 0x7fe4b8138920 con 0x7fe4b813f710 2024-03-25T19:34:01.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.426+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe49c004eb0 con 0x7fe4b813f710 2024-03-25T19:34:01.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.426+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fe4a80018f0 con 0x7fe4b813f710 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.426+0000 7fe4adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.426+0000 7fe4adffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.426+0000 7fe4adffb640 10 monclient(hunting): dump: 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:01.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.427+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1546932043 0 0) 0x7fe4b8138ab0 con 0x7fe4b81430c0 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.427+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b8138920 con 0x7fe4b81430c0 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.427+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fe4b0001b50 con 0x7fe4b81430c0 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.427+0000 7fe4adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.427+0000 7fe4adffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.427+0000 7fe4adffb640 10 monclient(hunting): dump: 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:01.428 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:01.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3674678799 0 0) 0x7fe4b80ab9e0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 >> v1:172.21.15.179:6789/0 conn(0x7fe4b81430c0 legacy=0x7fe4b81e1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:01.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 >> v1:172.21.15.52:6789/0 conn(0x7fe4b813f710 legacy=0x7fe4b81dacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:01.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4adffb640 10 monclient: _finish_hunting 0 2024-03-25T19:34:01.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4adffb640 1 monclient: found mon.c 2024-03-25T19:34:01.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:01.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b8137670 con 0x7fe4b80a98f0 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4adffb640 10 monclient: _finish_auth 0 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4adffb640 10 monclient: _check_auth_tickets 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4bfa0d640 5 monclient: authenticate success, global_id 4514 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4bfa0d640 10 monclient: _renew_subs 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4bfa0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/3676169040 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b81db3c0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4bfa0d640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4bfa0d640 10 monclient: _renew_subs 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4bfa0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/3676169040 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b81db710 con 0x7fe4b80a98f0 2024-03-25T19:34:01.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.435+0000 7fe4bfa0d640 1 librados: init done 2024-03-25T19:34:01.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.436+0000 7fe4bfa0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:01.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.436+0000 7fe4bfa0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:01.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.436+0000 7fe4bfa0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:01.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.436+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/3676169040 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b81378c0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.762+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b40038a0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.762+0000 7fe4adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:01.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.762+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fe4b4001250 con 0x7fe4b80a98f0 2024-03-25T19:34:01.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.762+0000 7fe4adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:01.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.762+0000 7fe4adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:01.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.762+0000 7fe4adffb640 10 monclient: dump: 2024-03-25T19:34:01.762 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:01.762 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:01.763 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:01.763 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:01.763 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:01.763 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:01.763 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:01.763 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:01.763 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:01.763 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:01.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.762+0000 7fe4af7fe640 4 set_mon_vals no callback set 2024-03-25T19:34:01.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.763+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fe4b4003d70 con 0x7fe4b80a98f0 2024-03-25T19:34:01.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.764+0000 7fe4bcfa8640 10 monclient: get_auth_request con 0x7fe49c06e0f0 auth_method 0 2024-03-25T19:34:01.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.764+0000 7fe4adffb640 10 client.4514.objecter ms_handle_connect 0x7fe49c06e0f0 2024-03-25T19:34:01.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.770+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fe4b40548b0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.770+0000 7fe4adffb640 10 client.4514.objecter ms_dispatch 0x7fe4b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:01.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.770+0000 7fe4adffb640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:01.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.770+0000 7fe4adffb640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:01.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.770+0000 7fe4adffb640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-25T19:34:01.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.770+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4b40503c0 con 0x7fe4b80a98f0 2024-03-25T19:34:01.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.770+0000 7fe4adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:01.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:01.770+0000 7fe4adffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:02.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.034+0000 7fe4bfa0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:02.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.034+0000 7fe4bfa0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:02.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.034+0000 7fe4bfa0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:02.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.034+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/3676169040 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4b80627b0 con 0x7fe4b80a98f0 2024-03-25T19:34:02.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.067+0000 7fe4adffb640 1 -- 172.21.15.52:0/3676169040 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fe4b40522f0 con 0x7fe4b80a98f0 2024-03-25T19:34:02.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.068+0000 7fe4adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:02.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.068+0000 7fe4adffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:02.067 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:34:02.068 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:02.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 10 librados: watch_flush enter 2024-03-25T19:34:02.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 10 librados: watch_flush exit 2024-03-25T19:34:02.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-25T19:34:02.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 10 client.4514.objecter successfully canceled tick 2024-03-25T19:34:02.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/3676169040 >> v1:172.21.15.52:6800/3967427147 conn(0x7fe49c06e0f0 legacy=0x7fe49c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 10 monclient: shutdown 2024-03-25T19:34:02.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:02.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/3676169040 >> v1:172.21.15.52:6790/0 conn(0x7fe4b80a98f0 legacy=0x7fe4b8138210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:02.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/3676169040 shutdown_connections 2024-03-25T19:34:02.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/3676169040 >> 172.21.15.52:0/3676169040 conn(0x7fe4b80b7ea0 msgr2=0x7fe4b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:02.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/3676169040 shutdown_connections 2024-03-25T19:34:02.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 1 -- 172.21.15.52:0/3676169040 wait complete. 2024-03-25T19:34:02.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:02.070+0000 7fe4bfa0d640 1 librados: shutdown 2024-03-25T19:34:02.081 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:08.888 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:08.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.000+0000 7f46af360640 10 monclient: get_monmap_and_config 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.000+0000 7f46af360640 10 monclient: build_initial_monmap 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.000+0000 7f46af360640 10 monclient: monmap: 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:34:09.000762+0000 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:34:09.000762+0000 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:34:09.000 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:09.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.000+0000 7f46af360640 1 Processor -- start 2024-03-25T19:34:09.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 1 -- start start 2024-03-25T19:34:09.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 10 monclient: init 2024-03-25T19:34:09.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:09.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:09.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 10 monclient(hunting): picked mon.noname-c con 0x7f46a80a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:09.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 10 monclient(hunting): picked mon.noname-a con 0x7f46a813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:09.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 10 monclient(hunting): picked mon.noname-b con 0x7f46a81430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:09.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a8142520 con 0x7f46a813f710 2024-03-25T19:34:09.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a80b28a0 con 0x7f46a80a98f0 2024-03-25T19:34:09.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a80b2a30 con 0x7f46a81430c0 2024-03-25T19:34:09.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:09.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.001+0000 7f46af360640 10 monclient(hunting): authenticate will time out at 6297.181568s 2024-03-25T19:34:09.006 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.006+0000 7f46ad8fd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f46a81430c0 0x7f46a814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:56882/0 (socket says 172.21.15.52:56882) 2024-03-25T19:34:09.006 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.006+0000 7f46ad8fd640 1 -- 172.21.15.52:0/1177057261 learned_addr learned my addr 172.21.15.52:0/1177057261 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:09.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.021+0000 7f469ffff640 1 -- 172.21.15.52:0/1177057261 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658331768 0 0) 0x7f46a80b2a30 con 0x7f46a81430c0 2024-03-25T19:34:09.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.021+0000 7f469ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:09.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.021+0000 7f469ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:09.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.021+0000 7f469ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:09.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.021+0000 7f469ffff640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:34:09.021 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.021+0000 7f469ffff640 1 -- 172.21.15.52:0/1177057261 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4694003520 con 0x7f46a81430c0 2024-03-25T19:34:09.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.023+0000 7f469ffff640 1 -- 172.21.15.52:0/1177057261 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942349910 0 0) 0x7f46a80b28a0 con 0x7f46a80a98f0 2024-03-25T19:34:09.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.023+0000 7f469ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:09.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.023+0000 7f469ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:09.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.023+0000 7f469ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:09.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.023+0000 7f469ffff640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:34:09.023 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.023+0000 7f469ffff640 1 -- 172.21.15.52:0/1177057261 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a80b2a30 con 0x7f46a80a98f0 2024-03-25T19:34:09.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 1 -- 172.21.15.52:0/1177057261 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2234291918 0 0) 0x7f4694003520 con 0x7f46a81430c0 2024-03-25T19:34:09.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 1 -- 172.21.15.52:0/1177057261 >> v1:172.21.15.179:6789/0 conn(0x7f46a80a98f0 legacy=0x7f46a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 1 -- 172.21.15.52:0/1177057261 >> v1:172.21.15.52:6789/0 conn(0x7f46a813f710 legacy=0x7f46a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:09.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 1 monclient: found mon.noname-b 2024-03-25T19:34:09.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:09.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:34:09.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 1 -- 172.21.15.52:0/1177057261 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a80b44c0 con 0x7f46a81430c0 2024-03-25T19:34:09.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 10 monclient: _finish_auth 0 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 1 -- 172.21.15.52:0/1177057261 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46a0001a10 con 0x7f46a81430c0 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f469ffff640 10 monclient: dump: 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:09.046 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:09.047 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:09.047 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:09.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f46af360640 5 monclient: authenticate success, global_id 4517 2024-03-25T19:34:09.047 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.045+0000 7f46af360640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:09.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.074+0000 7f469ffff640 1 -- 172.21.15.52:0/1177057261 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46a0001b70 con 0x7f46a81430c0 2024-03-25T19:34:09.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.074+0000 7f469ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:09.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.074+0000 7f469ffff640 1 -- 172.21.15.52:0/1177057261 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46a0001f80 con 0x7f46a81430c0 2024-03-25T19:34:09.074 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.074+0000 7f469ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.074+0000 7f469ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.074+0000 7f469ffff640 10 monclient: dump: 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.075+0000 7f46af360640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:09.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.075+0000 7f46af360640 4 set_mon_vals no callback set 2024-03-25T19:34:09.077 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.075+0000 7f46af360640 10 monclient: shutdown 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.075+0000 7f46af360640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.075+0000 7f46af360640 1 -- 172.21.15.52:0/1177057261 >> v1:172.21.15.52:6790/0 conn(0x7f46a81430c0 legacy=0x7f46a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.076+0000 7f46af360640 1 -- 172.21.15.52:0/1177057261 shutdown_connections 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.076+0000 7f46af360640 1 -- 172.21.15.52:0/1177057261 >> 172.21.15.52:0/1177057261 conn(0x7f46a80b7ea0 msgr2=0x7f46a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.076+0000 7f46af360640 1 -- 172.21.15.52:0/1177057261 shutdown_connections 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.076+0000 7f46af360640 1 -- 172.21.15.52:0/1177057261 wait complete. 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.077+0000 7f46af360640 10 monclient: build_initial_monmap 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.077+0000 7f46af360640 10 monclient: monmap: 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:34:09.078 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.077+0000 7f46af360640 1 librados: starting msgr at 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.077+0000 7f46af360640 1 librados: starting objecter 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.077+0000 7f46af360640 1 Processor -- start 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.077+0000 7f46af360640 1 -- start start 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.077+0000 7f46af360640 1 librados: setting wanted keys 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.077+0000 7f46af360640 1 librados: calling monclient init 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.077+0000 7f46af360640 10 monclient: init 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.078+0000 7f46af360640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:09.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.078+0000 7f46af360640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.078+0000 7f46af360640 10 monclient(hunting): picked mon.noname-b con 0x7f46a80a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.078+0000 7f46af360640 10 monclient(hunting): picked mon.noname-c con 0x7f46a813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.078+0000 7f46af360640 10 monclient(hunting): picked mon.noname-a con 0x7f46a81430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.078+0000 7f46af360640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a8136b50 con 0x7f46a81430c0 2024-03-25T19:34:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.078+0000 7f46af360640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a8135100 con 0x7f46a80a98f0 2024-03-25T19:34:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.078+0000 7f46af360640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a80b7ad0 con 0x7f46a813f710 2024-03-25T19:34:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.078+0000 7f46af360640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:09.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.078+0000 7f46af360640 10 monclient(hunting): authenticate will time out at 6297.258324s 2024-03-25T19:34:09.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.088+0000 7f46ad0fc640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f46a80a98f0 0x7f46a81d7670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:48434/0 (socket says 172.21.15.52:48434) 2024-03-25T19:34:09.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.088+0000 7f46ad0fc640 1 -- 172.21.15.52:0/1349156123 learned_addr learned my addr 172.21.15.52:0/1349156123 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:09.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.089+0000 7f469dffb640 10 client.?.objecter ms_handle_connect 0x7f46a80a98f0 2024-03-25T19:34:09.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.089+0000 7f469dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:09.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.092+0000 7f469dffb640 10 client.?.objecter ms_handle_connect 0x7f46a81430c0 2024-03-25T19:34:09.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.092+0000 7f469dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:09.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.098+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126677881 0 0) 0x7f46a8135100 con 0x7f46a80a98f0 2024-03-25T19:34:09.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.098+0000 7f469dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:09.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.098+0000 7f469dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:09.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.098+0000 7f469dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:09.098 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.098+0000 7f469dffb640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:34:09.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.098+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f468c004eb0 con 0x7f46a80a98f0 2024-03-25T19:34:09.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.099+0000 7f469dffb640 10 client.?.objecter ms_handle_connect 0x7f46a813f710 2024-03-25T19:34:09.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.099+0000 7f469dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:09.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.099+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352206421 0 0) 0x7f46a8136b50 con 0x7f46a81430c0 2024-03-25T19:34:09.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.099+0000 7f469dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:09.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.099+0000 7f469dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:09.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.099+0000 7f469dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:09.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.099+0000 7f469dffb640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:34:09.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.099+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a8135100 con 0x7f46a81430c0 2024-03-25T19:34:09.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.108+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657272124 0 0) 0x7f46a8135100 con 0x7f46a81430c0 2024-03-25T19:34:09.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.108+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46a8136b50 con 0x7f46a81430c0 2024-03-25T19:34:09.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.110+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46a0001e80 con 0x7f46a81430c0 2024-03-25T19:34:09.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.110+0000 7f469dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:09.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.110+0000 7f469dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.110+0000 7f469dffb640 10 monclient(hunting): dump: 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:09.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.110+0000 7f469dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:09.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1241284767 0 0) 0x7f46a8136b50 con 0x7f46a81430c0 2024-03-25T19:34:09.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 >> v1:172.21.15.52:6790/0 conn(0x7f46a813f710 legacy=0x7f46a8130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 >> v1:172.21.15.179:6789/0 conn(0x7f46a80a98f0 legacy=0x7f46a81d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f469dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f469dffb640 1 monclient: found mon.a 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f469dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f469dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a80b7c60 con 0x7f46a81430c0 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f469dffb640 10 monclient: _finish_auth 0 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f469dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f469dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f46af360640 5 monclient: authenticate success, global_id 4527 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f46af360640 10 monclient: _renew_subs 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f46af360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f46af360640 1 -- 172.21.15.52:0/1349156123 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a81d7d80 con 0x7f46a81430c0 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f46af360640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f46af360640 10 monclient: _renew_subs 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.115+0000 7f46af360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:09.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.116+0000 7f46af360640 1 -- 172.21.15.52:0/1349156123 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a813b2e0 con 0x7f46a81430c0 2024-03-25T19:34:09.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.116+0000 7f46af360640 1 librados: init done 2024-03-25T19:34:09.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.116+0000 7f46af360640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:09.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.116+0000 7f46af360640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:09.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.116+0000 7f46af360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:09.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.116+0000 7f46af360640 1 -- 172.21.15.52:0/1349156123 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a81d7f70 con 0x7f46a81430c0 2024-03-25T19:34:09.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.434+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46a0001560 con 0x7f46a81430c0 2024-03-25T19:34:09.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.434+0000 7f469dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:09.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.434+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46a0004e10 con 0x7f46a81430c0 2024-03-25T19:34:09.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.434+0000 7f469dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:09.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.434+0000 7f469dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:09.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.434+0000 7f469dffb640 10 monclient: dump: 2024-03-25T19:34:09.434 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:09.434 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:09.435 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:09.435 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:09.435 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:09.435 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:09.435 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:09.435 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:09.435 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:09.435 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:09.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.434+0000 7f469f7fe640 4 set_mon_vals no callback set 2024-03-25T19:34:09.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.435+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f46a001ae60 con 0x7f46a81430c0 2024-03-25T19:34:09.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.435+0000 7f46ad0fc640 10 monclient: get_auth_request con 0x7f468c06dcb0 auth_method 0 2024-03-25T19:34:09.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.436+0000 7f469dffb640 10 client.4527.objecter ms_handle_connect 0x7f468c06dcb0 2024-03-25T19:34:09.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.441+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f46a0054b70 con 0x7f46a81430c0 2024-03-25T19:34:09.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.441+0000 7f469dffb640 10 client.4527.objecter ms_dispatch 0x7f46a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:09.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.441+0000 7f469dffb640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:09.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.441+0000 7f469dffb640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:09.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.441+0000 7f469dffb640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-25T19:34:09.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.441+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46a00504b0 con 0x7f46a81430c0 2024-03-25T19:34:09.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.441+0000 7f469dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:09.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.441+0000 7f469dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:09.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.716+0000 7f46af360640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:09.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.716+0000 7f46af360640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:09.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.716+0000 7f46af360640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:09.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.716+0000 7f46af360640 1 -- 172.21.15.52:0/1349156123 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46a80627b0 con 0x7f46a81430c0 2024-03-25T19:34:09.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.762+0000 7f469dffb640 1 -- 172.21.15.52:0/1349156123 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f46a00525b0 con 0x7f46a81430c0 2024-03-25T19:34:09.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.762+0000 7f469dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:09.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.762+0000 7f469dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:09.762 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:34:09.762 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.764+0000 7f46af360640 10 librados: watch_flush enter 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.764+0000 7f46af360640 10 librados: watch_flush exit 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.764+0000 7f46af360640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.764+0000 7f46af360640 10 client.4527.objecter successfully canceled tick 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.765+0000 7f46af360640 1 -- 172.21.15.52:0/1349156123 >> v1:172.21.15.52:6800/3967427147 conn(0x7f468c06dcb0 legacy=0x7f468c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.765+0000 7f46af360640 10 monclient: shutdown 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.765+0000 7f46af360640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.765+0000 7f46af360640 1 -- 172.21.15.52:0/1349156123 >> v1:172.21.15.52:6789/0 conn(0x7f46a81430c0 legacy=0x7f46a81e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.765+0000 7f46af360640 1 -- 172.21.15.52:0/1349156123 shutdown_connections 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.765+0000 7f46af360640 1 -- 172.21.15.52:0/1349156123 >> 172.21.15.52:0/1349156123 conn(0x7f46a80b7ea0 msgr2=0x7f46a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.765+0000 7f46af360640 1 -- 172.21.15.52:0/1349156123 shutdown_connections 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.765+0000 7f46af360640 1 -- 172.21.15.52:0/1349156123 wait complete. 2024-03-25T19:34:09.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:09.765+0000 7f46af360640 1 librados: shutdown 2024-03-25T19:34:09.777 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:12.240 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:07:05.839 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.249 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:07:05.848 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.299 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:07:05.841 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.300 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:07:05.842 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.321 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:07:05.920 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.341 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:07:05.883 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.384 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:07:05.982 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.396 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:07:05.995 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:12.431 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:07:06.030 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:16.584 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:16.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.700+0000 7f108337c640 10 monclient: get_monmap_and_config 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.700+0000 7f108337c640 10 monclient: build_initial_monmap 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.700+0000 7f108337c640 10 monclient: monmap: 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:34:16.701450+0000 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:34:16.701450+0000 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:34:16.701 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:16.702 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.701+0000 7f108337c640 1 Processor -- start 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.701+0000 7f108337c640 1 -- start start 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.701+0000 7f108337c640 10 monclient: init 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.702+0000 7f108337c640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.702+0000 7f108337c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.702+0000 7f108337c640 10 monclient(hunting): picked mon.noname-b con 0x7f107c0a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.702+0000 7f108337c640 10 monclient(hunting): picked mon.noname-c con 0x7f107c13f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.702+0000 7f108337c640 10 monclient(hunting): picked mon.noname-a con 0x7f107c1430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.702+0000 7f108337c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c142520 con 0x7f107c1430c0 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.702+0000 7f108337c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c0b28a0 con 0x7f107c13f710 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.702+0000 7f108337c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c0b2a30 con 0x7f107c0a98f0 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.702+0000 7f108337c640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:16.703 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.702+0000 7f108337c640 10 monclient(hunting): authenticate will time out at 6304.882443s 2024-03-25T19:34:16.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.709+0000 7f1080917640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f107c13f710 0x7f107c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:39962/0 (socket says 172.21.15.52:39962) 2024-03-25T19:34:16.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.709+0000 7f1080917640 1 -- 172.21.15.52:0/3312002430 learned_addr learned my addr 172.21.15.52:0/3312002430 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:16.722 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.723+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40769916 0 0) 0x7f107c0b28a0 con 0x7f107c13f710 2024-03-25T19:34:16.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.723+0000 7f1073fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:16.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.723+0000 7f1073fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:16.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.723+0000 7f1073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:16.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.723+0000 7f1073fff640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:34:16.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.723+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1068003520 con 0x7f107c13f710 2024-03-25T19:34:16.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.726+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910339055 0 0) 0x7f107c142520 con 0x7f107c1430c0 2024-03-25T19:34:16.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.726+0000 7f1073fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:16.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.726+0000 7f1073fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:16.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.726+0000 7f1073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:16.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.726+0000 7f1073fff640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:34:16.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.726+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107c0b28a0 con 0x7f107c1430c0 2024-03-25T19:34:16.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.728+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647572232 0 0) 0x7f107c0b2a30 con 0x7f107c0a98f0 2024-03-25T19:34:16.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.728+0000 7f1073fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:16.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.728+0000 7f1073fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:16.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.728+0000 7f1073fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:16.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.728+0000 7f1073fff640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:34:16.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.728+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107c142520 con 0x7f107c0a98f0 2024-03-25T19:34:16.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1576105920 0 0) 0x7f107c0b28a0 con 0x7f107c1430c0 2024-03-25T19:34:16.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 >> v1:172.21.15.52:6790/0 conn(0x7f107c0a98f0 legacy=0x7f107c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:16.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 >> v1:172.21.15.179:6789/0 conn(0x7f107c13f710 legacy=0x7f107c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:16.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:16.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 1 monclient: found mon.noname-a 2024-03-25T19:34:16.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:16.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:34:16.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c0b44c0 con 0x7f107c1430c0 2024-03-25T19:34:16.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 10 monclient: _finish_auth 0 2024-03-25T19:34:16.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:16.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f10740019f0 con 0x7f107c1430c0 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f1073fff640 10 monclient: dump: 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f108337c640 5 monclient: authenticate success, global_id 4530 2024-03-25T19:34:16.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.741+0000 7f108337c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:16.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1074001b50 con 0x7f107c1430c0 2024-03-25T19:34:16.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f1073fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:16.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f1073fff640 1 -- 172.21.15.52:0/3312002430 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1074001f60 con 0x7f107c1430c0 2024-03-25T19:34:16.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f108337c640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:16.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f108337c640 4 set_mon_vals no callback set 2024-03-25T19:34:16.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f1073fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:16.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f1073fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:16.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f1073fff640 10 monclient: dump: 2024-03-25T19:34:16.756 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:16.756 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:16.756 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:16.756 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:16.757 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:16.757 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:16.757 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:16.757 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:16.757 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:16.757 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:16.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f108337c640 10 monclient: shutdown 2024-03-25T19:34:16.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f108337c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f108337c640 1 -- 172.21.15.52:0/3312002430 >> v1:172.21.15.52:6789/0 conn(0x7f107c1430c0 legacy=0x7f107c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f108337c640 1 -- 172.21.15.52:0/3312002430 shutdown_connections 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.756+0000 7f108337c640 1 -- 172.21.15.52:0/3312002430 >> 172.21.15.52:0/3312002430 conn(0x7f107c0b7ea0 msgr2=0x7f107c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.757+0000 7f108337c640 1 -- 172.21.15.52:0/3312002430 shutdown_connections 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.757+0000 7f108337c640 1 -- 172.21.15.52:0/3312002430 wait complete. 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.757+0000 7f108337c640 10 monclient: build_initial_monmap 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.757+0000 7f108337c640 10 monclient: monmap: 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:34:16.760 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 1 librados: starting msgr at 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 1 librados: starting objecter 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 1 Processor -- start 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 1 -- start start 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 1 librados: setting wanted keys 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 1 librados: calling monclient init 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 10 monclient: init 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 10 monclient(hunting): picked mon.noname-b con 0x7f107c0a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 10 monclient(hunting): picked mon.noname-a con 0x7f107c13f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 10 monclient(hunting): picked mon.noname-c con 0x7f107c1430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:16.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c139920 con 0x7f107c13f710 2024-03-25T19:34:16.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c1db380 con 0x7f107c0a98f0 2024-03-25T19:34:16.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c1db620 con 0x7f107c1430c0 2024-03-25T19:34:16.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:16.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.758+0000 7f108337c640 10 monclient(hunting): authenticate will time out at 6304.938845s 2024-03-25T19:34:16.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.763+0000 7f1081118640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f107c0a98f0 0x7f107c1dac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:39998/0 (socket says 172.21.15.52:39998) 2024-03-25T19:34:16.762 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.763+0000 7f1081118640 1 -- 172.21.15.52:0/2022168802 learned_addr learned my addr 172.21.15.52:0/2022168802 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:16.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.764+0000 7f1071ffb640 10 client.?.objecter ms_handle_connect 0x7f107c0a98f0 2024-03-25T19:34:16.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.764+0000 7f1071ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:16.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.767+0000 7f1071ffb640 10 client.?.objecter ms_handle_connect 0x7f107c1430c0 2024-03-25T19:34:16.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.767+0000 7f1071ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:16.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.771+0000 7f1071ffb640 10 client.?.objecter ms_handle_connect 0x7f107c13f710 2024-03-25T19:34:16.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.771+0000 7f1071ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:16.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.779+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986965546 0 0) 0x7f107c139920 con 0x7f107c13f710 2024-03-25T19:34:16.778 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.779+0000 7f1071ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:16.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.779+0000 7f1071ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:16.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.779+0000 7f1071ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:16.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.779+0000 7f1071ffb640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:34:16.779 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.779+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1060004eb0 con 0x7f107c13f710 2024-03-25T19:34:16.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.784+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743028679 0 0) 0x7f107c1db620 con 0x7f107c1430c0 2024-03-25T19:34:16.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.784+0000 7f1071ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:16.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.784+0000 7f1071ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:16.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.784+0000 7f1071ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:16.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.784+0000 7f1071ffb640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:34:16.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.784+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107c139920 con 0x7f107c1430c0 2024-03-25T19:34:16.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.787+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490219508 0 0) 0x7f107c1db380 con 0x7f107c0a98f0 2024-03-25T19:34:16.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.787+0000 7f1071ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:16.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.787+0000 7f1071ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:16.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.787+0000 7f1071ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:16.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.787+0000 7f1071ffb640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:34:16.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.787+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107c1db620 con 0x7f107c0a98f0 2024-03-25T19:34:16.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.796+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4117333443 0 0) 0x7f1060004eb0 con 0x7f107c13f710 2024-03-25T19:34:16.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.796+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f107c1db380 con 0x7f107c13f710 2024-03-25T19:34:16.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.796+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f106c001950 con 0x7f107c13f710 2024-03-25T19:34:16.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.796+0000 7f1071ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:16.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.796+0000 7f1071ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:16.796 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.796+0000 7f1071ffb640 10 monclient(hunting): dump: 2024-03-25T19:34:16.796 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:16.796 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:16.796 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:16.796 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:16.797 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:16.797 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:16.797 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:16.797 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:16.797 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:16.797 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:16.797 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.796+0000 7f1071ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:16.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.801+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2483014333 0 0) 0x7f107c1db620 con 0x7f107c0a98f0 2024-03-25T19:34:16.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.801+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1060004eb0 con 0x7f107c0a98f0 2024-03-25T19:34:16.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.801+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f10780019b0 con 0x7f107c0a98f0 2024-03-25T19:34:16.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.801+0000 7f1071ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:16.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.801+0000 7f1071ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:16.801 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.801+0000 7f1071ffb640 10 monclient(hunting): dump: 2024-03-25T19:34:16.802 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:16.802 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:16.802 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:16.802 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:16.802 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:16.802 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:16.802 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:16.802 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:16.802 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:16.802 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:16.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 360003850 0 0) 0x7f107c1db380 con 0x7f107c13f710 2024-03-25T19:34:16.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 >> v1:172.21.15.52:6790/0 conn(0x7f107c1430c0 legacy=0x7f107c137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:16.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 >> v1:172.21.15.179:6789/0 conn(0x7f107c0a98f0 legacy=0x7f107c1dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:16.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f1071ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:34:16.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f1071ffb640 1 monclient: found mon.a 2024-03-25T19:34:16.803 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f1071ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f1071ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c0b0dc0 con 0x7f107c13f710 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f1071ffb640 10 monclient: _finish_auth 0 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f1071ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f1071ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f108337c640 5 monclient: authenticate success, global_id 4533 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f108337c640 10 monclient: _renew_subs 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f108337c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f108337c640 1 -- 172.21.15.52:0/2022168802 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c0b1010 con 0x7f107c13f710 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f108337c640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f108337c640 10 monclient: _renew_subs 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f108337c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f108337c640 1 -- 172.21.15.52:0/2022168802 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c0b0910 con 0x7f107c13f710 2024-03-25T19:34:16.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.803+0000 7f108337c640 1 librados: init done 2024-03-25T19:34:16.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.804+0000 7f108337c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:16.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.804+0000 7f108337c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:16.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.804+0000 7f108337c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:16.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:16.804+0000 7f108337c640 1 -- 172.21.15.52:0/2022168802 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c1db7b0 con 0x7f107c13f710 2024-03-25T19:34:17.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.124+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f106c001d60 con 0x7f107c13f710 2024-03-25T19:34:17.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.124+0000 7f1071ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:17.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.124+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f106c004980 con 0x7f107c13f710 2024-03-25T19:34:17.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.124+0000 7f1071ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:17.124 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.124+0000 7f1071ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.124+0000 7f1071ffb640 10 monclient: dump: 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:17.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.124+0000 7f10737fe640 4 set_mon_vals no callback set 2024-03-25T19:34:17.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.125+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f106c004ba0 con 0x7f107c13f710 2024-03-25T19:34:17.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.125+0000 7f1081118640 10 monclient: get_auth_request con 0x7f106006df70 auth_method 0 2024-03-25T19:34:17.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.126+0000 7f1071ffb640 10 client.4533.objecter ms_handle_connect 0x7f106006df70 2024-03-25T19:34:17.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.131+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f106c054710 con 0x7f107c13f710 2024-03-25T19:34:17.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.131+0000 7f1071ffb640 10 client.4533.objecter ms_dispatch 0x7f107c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:17.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.131+0000 7f1071ffb640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:17.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.131+0000 7f1071ffb640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:17.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.131+0000 7f1071ffb640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-25T19:34:17.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.131+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f106c050220 con 0x7f107c13f710 2024-03-25T19:34:17.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.131+0000 7f1071ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:17.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.131+0000 7f1071ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:17.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.383+0000 7f108337c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:17.463 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.383+0000 7f108337c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:17.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.383+0000 7f108337c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:17.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.383+0000 7f108337c640 1 -- 172.21.15.52:0/2022168802 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f107c0b0bc0 con 0x7f107c13f710 2024-03-25T19:34:17.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.417+0000 7f1071ffb640 1 -- 172.21.15.52:0/2022168802 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f106c052150 con 0x7f107c13f710 2024-03-25T19:34:17.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.417+0000 7f1071ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:17.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.417+0000 7f1071ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:17.465 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:34:17.465 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:17.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.419+0000 7f108337c640 10 librados: watch_flush enter 2024-03-25T19:34:17.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.419+0000 7f108337c640 10 librados: watch_flush exit 2024-03-25T19:34:17.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.419+0000 7f108337c640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-25T19:34:17.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.419+0000 7f108337c640 10 client.4533.objecter successfully canceled tick 2024-03-25T19:34:17.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.420+0000 7f108337c640 1 -- 172.21.15.52:0/2022168802 >> v1:172.21.15.52:6800/3967427147 conn(0x7f106006df70 legacy=0x7f1060070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.420+0000 7f108337c640 10 monclient: shutdown 2024-03-25T19:34:17.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.420+0000 7f108337c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:17.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.420+0000 7f108337c640 1 -- 172.21.15.52:0/2022168802 >> v1:172.21.15.52:6789/0 conn(0x7f107c13f710 legacy=0x7f107c0b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:17.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.420+0000 7f108337c640 1 -- 172.21.15.52:0/2022168802 shutdown_connections 2024-03-25T19:34:17.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.420+0000 7f108337c640 1 -- 172.21.15.52:0/2022168802 >> 172.21.15.52:0/2022168802 conn(0x7f107c0b7ea0 msgr2=0x7f107c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:17.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.420+0000 7f108337c640 1 -- 172.21.15.52:0/2022168802 shutdown_connections 2024-03-25T19:34:17.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.420+0000 7f108337c640 1 -- 172.21.15.52:0/2022168802 wait complete. 2024-03-25T19:34:17.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:17.420+0000 7f108337c640 1 librados: shutdown 2024-03-25T19:34:17.468 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:24.274 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:24.392 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.393+0000 7f36bb6a8640 10 monclient: get_monmap_and_config 2024-03-25T19:34:24.392 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.393+0000 7f36bb6a8640 10 monclient: build_initial_monmap 2024-03-25T19:34:24.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.393+0000 7f36bb6a8640 10 monclient: monmap: 2024-03-25T19:34:24.393 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:24.393 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:24.393 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:34:24.393646+0000 2024-03-25T19:34:24.393 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:34:24.393646+0000 2024-03-25T19:34:24.393 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:24.393 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:24.393 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:24.393 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:34:24.393 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:34:24.393 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:24.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.393+0000 7f36bb6a8640 1 Processor -- start 2024-03-25T19:34:24.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 1 -- start start 2024-03-25T19:34:24.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 10 monclient: init 2024-03-25T19:34:24.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:24.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:24.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 10 monclient(hunting): picked mon.noname-c con 0x7f36b40a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:24.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 10 monclient(hunting): picked mon.noname-a con 0x7f36b413f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:24.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 10 monclient(hunting): picked mon.noname-b con 0x7f36b41430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:24.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b4142520 con 0x7f36b413f710 2024-03-25T19:34:24.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40b28a0 con 0x7f36b40a98f0 2024-03-25T19:34:24.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40b2a30 con 0x7f36b41430c0 2024-03-25T19:34:24.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:24.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.394+0000 7f36bb6a8640 10 monclient(hunting): authenticate will time out at 6312.574755s 2024-03-25T19:34:24.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.398+0000 7f36b8c43640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f36b413f710 0x7f36b41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:44116/0 (socket says 172.21.15.52:44116) 2024-03-25T19:34:24.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.398+0000 7f36b8c43640 1 -- 172.21.15.52:0/1668484095 learned_addr learned my addr 172.21.15.52:0/1668484095 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.404+0000 7f36abfff640 1 -- 172.21.15.52:0/1668484095 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571007028 0 0) 0x7f36b40b28a0 con 0x7f36b40a98f0 2024-03-25T19:34:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.404+0000 7f36abfff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.404+0000 7f36abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.404+0000 7f36abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.404+0000 7f36abfff640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:34:24.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.404+0000 7f36abfff640 1 -- 172.21.15.52:0/1668484095 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a0003520 con 0x7f36b40a98f0 2024-03-25T19:34:24.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.406+0000 7f36abfff640 1 -- 172.21.15.52:0/1668484095 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612781835 0 0) 0x7f36b4142520 con 0x7f36b413f710 2024-03-25T19:34:24.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.407+0000 7f36abfff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:24.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.407+0000 7f36abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:24.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.407+0000 7f36abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:24.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.407+0000 7f36abfff640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:34:24.406 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.407+0000 7f36abfff640 1 -- 172.21.15.52:0/1668484095 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b40b28a0 con 0x7f36b413f710 2024-03-25T19:34:24.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 1 -- 172.21.15.52:0/1668484095 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3655811787 0 0) 0x7f36b40b28a0 con 0x7f36b413f710 2024-03-25T19:34:24.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 1 -- 172.21.15.52:0/1668484095 >> v1:172.21.15.52:6790/0 conn(0x7f36b41430c0 legacy=0x7f36b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:24.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 1 -- 172.21.15.52:0/1668484095 >> v1:172.21.15.179:6789/0 conn(0x7f36b40a98f0 legacy=0x7f36b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:24.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:24.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 1 monclient: found mon.noname-a 2024-03-25T19:34:24.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 1 -- 172.21.15.52:0/1668484095 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b40b44c0 con 0x7f36b413f710 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 10 monclient: _finish_auth 0 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 1 -- 172.21.15.52:0/1668484095 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f36a4001670 con 0x7f36b413f710 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36bb6a8640 5 monclient: authenticate success, global_id 4536 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36bb6a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36abfff640 10 monclient: dump: 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:24.418 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:24.419 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:24.419 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:24.419 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:24.419 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:24.419 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:24.419 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.417+0000 7f36bb6a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:24.425 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.426+0000 7f36abfff640 1 -- 172.21.15.52:0/1668484095 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36a40017d0 con 0x7f36b413f710 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.426+0000 7f36abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.426+0000 7f36abfff640 1 -- 172.21.15.52:0/1668484095 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f36a4001ba0 con 0x7f36b413f710 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.426+0000 7f36bb6a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.426+0000 7f36bb6a8640 4 set_mon_vals no callback set 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.426+0000 7f36abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.426+0000 7f36abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.426+0000 7f36abfff640 10 monclient: dump: 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:24.426 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:24.427 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:24.427 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:24.427 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:24.427 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.427+0000 7f36bb6a8640 10 monclient: shutdown 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.427+0000 7f36bb6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.427+0000 7f36bb6a8640 1 -- 172.21.15.52:0/1668484095 >> v1:172.21.15.52:6789/0 conn(0x7f36b413f710 legacy=0x7f36b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.427+0000 7f36bb6a8640 1 -- 172.21.15.52:0/1668484095 shutdown_connections 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.427+0000 7f36bb6a8640 1 -- 172.21.15.52:0/1668484095 >> 172.21.15.52:0/1668484095 conn(0x7f36b40b7ea0 msgr2=0x7f36b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.427+0000 7f36bb6a8640 1 -- 172.21.15.52:0/1668484095 shutdown_connections 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.427+0000 7f36bb6a8640 1 -- 172.21.15.52:0/1668484095 wait complete. 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.428+0000 7f36bb6a8640 10 monclient: build_initial_monmap 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.428+0000 7f36bb6a8640 10 monclient: monmap: 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:34:24.429 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.428+0000 7f36bb6a8640 1 librados: starting msgr at 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.428+0000 7f36bb6a8640 1 librados: starting objecter 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.428+0000 7f36bb6a8640 1 Processor -- start 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.428+0000 7f36bb6a8640 1 -- start start 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.428+0000 7f36bb6a8640 1 librados: setting wanted keys 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.428+0000 7f36bb6a8640 1 librados: calling monclient init 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.428+0000 7f36bb6a8640 10 monclient: init 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.429+0000 7f36bb6a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:24.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.429+0000 7f36bb6a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:24.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.429+0000 7f36bb6a8640 10 monclient(hunting): picked mon.noname-c con 0x7f36b40a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:24.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.429+0000 7f36bb6a8640 10 monclient(hunting): picked mon.noname-b con 0x7f36b413f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:24.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.429+0000 7f36bb6a8640 10 monclient(hunting): picked mon.noname-a con 0x7f36b41430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:24.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.429+0000 7f36bb6a8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b40abc00 con 0x7f36b41430c0 2024-03-25T19:34:24.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.429+0000 7f36bb6a8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b41d51f0 con 0x7f36b413f710 2024-03-25T19:34:24.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.429+0000 7f36bb6a8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b4138c70 con 0x7f36b40a98f0 2024-03-25T19:34:24.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.429+0000 7f36bb6a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:24.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.429+0000 7f36bb6a8640 10 monclient(hunting): authenticate will time out at 6312.609484s 2024-03-25T19:34:24.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.435+0000 7f36b8c43640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f36b413f710 0x7f36b41d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:44786/0 (socket says 172.21.15.52:44786) 2024-03-25T19:34:24.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.435+0000 7f36b8c43640 1 -- 172.21.15.52:0/3535040480 learned_addr learned my addr 172.21.15.52:0/3535040480 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:24.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.437+0000 7f36a9ffb640 10 client.?.objecter ms_handle_connect 0x7f36b413f710 2024-03-25T19:34:24.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.437+0000 7f36a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:24.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.441+0000 7f36a9ffb640 10 client.?.objecter ms_handle_connect 0x7f36b40a98f0 2024-03-25T19:34:24.441 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.441+0000 7f36a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:24.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.447+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665428643 0 0) 0x7f36b41d51f0 con 0x7f36b413f710 2024-03-25T19:34:24.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.447+0000 7f36a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:24.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.447+0000 7f36a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:24.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.447+0000 7f36a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:24.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.447+0000 7f36a9ffb640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:34:24.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.447+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3698004eb0 con 0x7f36b413f710 2024-03-25T19:34:24.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.459+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3356558548 0 0) 0x7f36b4138c70 con 0x7f36b40a98f0 2024-03-25T19:34:24.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.459+0000 7f36a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:24.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.459+0000 7f36a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:24.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.459+0000 7f36a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:24.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.459+0000 7f36a9ffb640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:34:24.459 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.459+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b41d51f0 con 0x7f36b40a98f0 2024-03-25T19:34:24.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.461+0000 7f36a9ffb640 10 client.?.objecter ms_handle_connect 0x7f36b41430c0 2024-03-25T19:34:24.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.461+0000 7f36a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:24.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.466+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 595429107 0 0) 0x7f3698004eb0 con 0x7f36b413f710 2024-03-25T19:34:24.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.466+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36b4138c70 con 0x7f36b413f710 2024-03-25T19:34:24.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.466+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f36a4001ba0 con 0x7f36b413f710 2024-03-25T19:34:24.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.466+0000 7f36a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:24.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.466+0000 7f36a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:24.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.466+0000 7f36a9ffb640 10 monclient(hunting): dump: 2024-03-25T19:34:24.466 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:24.466 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:24.466 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:24.466 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:24.467 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:24.467 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:24.467 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:24.467 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:24.467 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:24.467 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:24.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.466+0000 7f36a9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:24.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.470+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582012893 0 0) 0x7f36b40abc00 con 0x7f36b41430c0 2024-03-25T19:34:24.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.470+0000 7f36a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:24.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.470+0000 7f36a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:24.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.470+0000 7f36a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:24.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.470+0000 7f36a9ffb640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:34:24.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.470+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3698004eb0 con 0x7f36b41430c0 2024-03-25T19:34:24.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 33618802 0 0) 0x7f36b4138c70 con 0x7f36b413f710 2024-03-25T19:34:24.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 >> v1:172.21.15.52:6790/0 conn(0x7f36b40a98f0 legacy=0x7f36b4138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:24.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 >> v1:172.21.15.52:6789/0 conn(0x7f36b41430c0 legacy=0x7f36b41e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:24.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36a9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:34:24.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36a9ffb640 1 monclient: found mon.b 2024-03-25T19:34:24.488 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36a9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b41d7730 con 0x7f36b413f710 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36a9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36a9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36bb6a8640 5 monclient: authenticate success, global_id 4546 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36bb6a8640 10 monclient: _renew_subs 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36bb6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36bb6a8640 1 -- 172.21.15.52:0/3535040480 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b41d8340 con 0x7f36b413f710 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36bb6a8640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36bb6a8640 10 monclient: _renew_subs 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36bb6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36bb6a8640 1 -- 172.21.15.52:0/3535040480 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b4137f00 con 0x7f36b413f710 2024-03-25T19:34:24.489 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.488+0000 7f36bb6a8640 1 librados: init done 2024-03-25T19:34:24.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.489+0000 7f36bb6a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:24.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.489+0000 7f36bb6a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:24.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.489+0000 7f36bb6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:34:24.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.489+0000 7f36bb6a8640 1 -- 172.21.15.52:0/3535040480 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b40627b0 con 0x7f36b413f710 2024-03-25T19:34:24.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.763+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36a4004a90 con 0x7f36b413f710 2024-03-25T19:34:24.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.763+0000 7f36a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:24.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.763+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f36a4004ea0 con 0x7f36b413f710 2024-03-25T19:34:24.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.763+0000 7f36a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:24.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.763+0000 7f36a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:24.763 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.763+0000 7f36a9ffb640 10 monclient: dump: 2024-03-25T19:34:24.764 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:24.764 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:24.764 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:24.764 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:24.764 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:24.764 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:24.764 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:24.764 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:24.764 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:24.764 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:24.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.763+0000 7f36ab7fe640 4 set_mon_vals no callback set 2024-03-25T19:34:24.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.764+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f36a40050c0 con 0x7f36b413f710 2024-03-25T19:34:24.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.765+0000 7f36b9444640 10 monclient: get_auth_request con 0x7f369806de00 auth_method 0 2024-03-25T19:34:24.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.765+0000 7f36a9ffb640 10 client.4546.objecter ms_handle_connect 0x7f369806de00 2024-03-25T19:34:24.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.769+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f36a40549b0 con 0x7f36b413f710 2024-03-25T19:34:24.768 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.769+0000 7f36a9ffb640 10 client.4546.objecter ms_dispatch 0x7f36b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:24.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.769+0000 7f36a9ffb640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:24.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.769+0000 7f36a9ffb640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:24.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.769+0000 7f36a9ffb640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T19:34:24.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.772+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36a40504c0 con 0x7f36b413f710 2024-03-25T19:34:24.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.772+0000 7f36a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:24.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.772+0000 7f36a9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:24.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.990+0000 7f36bb6a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:24.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.990+0000 7f36bb6a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:24.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.990+0000 7f36bb6a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:34:24.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:24.990+0000 7f36bb6a8640 1 -- 172.21.15.52:0/3535040480 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36b41d78c0 con 0x7f36b413f710 2024-03-25T19:34:25.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.111+0000 7f36a9ffb640 1 -- 172.21.15.52:0/3535040480 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f36a40523f0 con 0x7f36b413f710 2024-03-25T19:34:25.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.111+0000 7f36a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:25.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.111+0000 7f36a9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:25.111 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:34:25.112 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:25.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 10 librados: watch_flush enter 2024-03-25T19:34:25.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 10 librados: watch_flush exit 2024-03-25T19:34:25.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T19:34:25.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 10 client.4546.objecter successfully canceled tick 2024-03-25T19:34:25.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 1 -- 172.21.15.52:0/3535040480 >> v1:172.21.15.52:6800/3967427147 conn(0x7f369806de00 legacy=0x7f36980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 10 monclient: shutdown 2024-03-25T19:34:25.114 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:25.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 1 -- 172.21.15.52:0/3535040480 >> v1:172.21.15.179:6789/0 conn(0x7f36b413f710 legacy=0x7f36b41d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:25.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 1 -- 172.21.15.52:0/3535040480 shutdown_connections 2024-03-25T19:34:25.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 1 -- 172.21.15.52:0/3535040480 >> 172.21.15.52:0/3535040480 conn(0x7f36b40b7ea0 msgr2=0x7f36b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:25.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 1 -- 172.21.15.52:0/3535040480 shutdown_connections 2024-03-25T19:34:25.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 1 -- 172.21.15.52:0/3535040480 wait complete. 2024-03-25T19:34:25.115 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:25.114+0000 7f36bb6a8640 1 librados: shutdown 2024-03-25T19:34:25.125 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:31.932 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:32.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.049+0000 7fb9a8c45640 10 monclient: get_monmap_and_config 2024-03-25T19:34:32.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.049+0000 7fb9a8c45640 10 monclient: build_initial_monmap 2024-03-25T19:34:32.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.049+0000 7fb9a8c45640 10 monclient: monmap: 2024-03-25T19:34:32.049 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:32.050 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:32.050 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:34:32.050743+0000 2024-03-25T19:34:32.050 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:34:32.050743+0000 2024-03-25T19:34:32.050 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:32.050 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:32.050 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:32.050 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:34:32.050 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:34:32.050 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:32.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.049+0000 7fb9a8c45640 1 Processor -- start 2024-03-25T19:34:32.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.049+0000 7fb9a8c45640 1 -- start start 2024-03-25T19:34:32.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.049+0000 7fb9a8c45640 10 monclient: init 2024-03-25T19:34:32.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.050+0000 7fb9a8c45640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:32.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.050+0000 7fb9a8c45640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:32.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.050+0000 7fb9a8c45640 10 monclient(hunting): picked mon.noname-a con 0x7fb9a40a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:32.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.050+0000 7fb9a8c45640 10 monclient(hunting): picked mon.noname-c con 0x7fb9a413f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:32.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.050+0000 7fb9a8c45640 10 monclient(hunting): picked mon.noname-b con 0x7fb9a41430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:32.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.050+0000 7fb9a8c45640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a4142520 con 0x7fb9a40a98f0 2024-03-25T19:34:32.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.050+0000 7fb9a8c45640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a40b28a0 con 0x7fb9a413f710 2024-03-25T19:34:32.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.050+0000 7fb9a8c45640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a40b2a30 con 0x7fb9a41430c0 2024-03-25T19:34:32.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.050+0000 7fb9a8c45640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:32.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.050+0000 7fb9a8c45640 10 monclient(hunting): authenticate will time out at 6320.231418s 2024-03-25T19:34:32.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.057+0000 7fb9a2d9d640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fb9a41430c0 0x7fb9a414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:49032/0 (socket says 172.21.15.52:49032) 2024-03-25T19:34:32.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.057+0000 7fb9a2d9d640 1 -- 172.21.15.52:0/3907604022 learned_addr learned my addr 172.21.15.52:0/3907604022 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:32.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.065+0000 7fb9a1d9b640 1 -- 172.21.15.52:0/3907604022 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519009409 0 0) 0x7fb9a40b2a30 con 0x7fb9a41430c0 2024-03-25T19:34:32.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.065+0000 7fb9a1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.065+0000 7fb9a1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.065+0000 7fb9a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.065+0000 7fb9a1d9b640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:34:32.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.065+0000 7fb9a1d9b640 1 -- 172.21.15.52:0/3907604022 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb990003520 con 0x7fb9a41430c0 2024-03-25T19:34:32.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.074+0000 7fb9a1d9b640 1 -- 172.21.15.52:0/3907604022 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515017429 0 0) 0x7fb9a4142520 con 0x7fb9a40a98f0 2024-03-25T19:34:32.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.074+0000 7fb9a1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.074+0000 7fb9a1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.074+0000 7fb9a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.074+0000 7fb9a1d9b640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:34:32.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.074+0000 7fb9a1d9b640 1 -- 172.21.15.52:0/3907604022 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a40b2a30 con 0x7fb9a40a98f0 2024-03-25T19:34:32.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 1 -- 172.21.15.52:0/3907604022 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1365033841 0 0) 0x7fb9a40b2a30 con 0x7fb9a40a98f0 2024-03-25T19:34:32.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 1 -- 172.21.15.52:0/3907604022 >> v1:172.21.15.52:6790/0 conn(0x7fb9a41430c0 legacy=0x7fb9a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.081 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 1 -- 172.21.15.52:0/3907604022 >> v1:172.21.15.179:6789/0 conn(0x7fb9a413f710 legacy=0x7fb9a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 1 monclient: found mon.noname-a 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 1 -- 172.21.15.52:0/3907604022 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a40b44c0 con 0x7fb9a40a98f0 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 10 monclient: _finish_auth 0 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 1 -- 172.21.15.52:0/3907604022 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb994001670 con 0x7fb9a40a98f0 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a1d9b640 10 monclient: dump: 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:32.082 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:32.083 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.083 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:32.083 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:32.083 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:32.083 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:32.083 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:32.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.080+0000 7fb9a8c45640 5 monclient: authenticate success, global_id 4542 2024-03-25T19:34:32.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.081+0000 7fb9a8c45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:32.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.085+0000 7fb9a1d9b640 1 -- 172.21.15.52:0/3907604022 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9940017d0 con 0x7fb9a40a98f0 2024-03-25T19:34:32.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.085+0000 7fb9a1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:32.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.085+0000 7fb9a1d9b640 1 -- 172.21.15.52:0/3907604022 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb994001c20 con 0x7fb9a40a98f0 2024-03-25T19:34:32.085 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.085+0000 7fb9a1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.085+0000 7fb9a1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.085+0000 7fb9a1d9b640 10 monclient: dump: 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:32.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.085+0000 7fb9a8c45640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:32.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.085+0000 7fb9a8c45640 4 set_mon_vals no callback set 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.086+0000 7fb9a8c45640 10 monclient: shutdown 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.086+0000 7fb9a8c45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.086+0000 7fb9a8c45640 1 -- 172.21.15.52:0/3907604022 >> v1:172.21.15.52:6789/0 conn(0x7fb9a40a98f0 legacy=0x7fb9a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.086+0000 7fb9a8c45640 1 -- 172.21.15.52:0/3907604022 shutdown_connections 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.086+0000 7fb9a8c45640 1 -- 172.21.15.52:0/3907604022 >> 172.21.15.52:0/3907604022 conn(0x7fb9a40b7ea0 msgr2=0x7fb9a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.086+0000 7fb9a8c45640 1 -- 172.21.15.52:0/3907604022 shutdown_connections 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.086+0000 7fb9a8c45640 1 -- 172.21.15.52:0/3907604022 wait complete. 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.087+0000 7fb9a8c45640 10 monclient: build_initial_monmap 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.087+0000 7fb9a8c45640 10 monclient: monmap: 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:34:32.088 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:34:32.089 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:32.089 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:34:32.089 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:32.089 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:34:32.089 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:34:32.089 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:32.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.087+0000 7fb9a8c45640 1 librados: starting msgr at 2024-03-25T19:34:32.089 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.087+0000 7fb9a8c45640 1 librados: starting objecter 2024-03-25T19:34:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 1 Processor -- start 2024-03-25T19:34:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 1 -- start start 2024-03-25T19:34:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 1 librados: setting wanted keys 2024-03-25T19:34:32.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 1 librados: calling monclient init 2024-03-25T19:34:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 10 monclient: init 2024-03-25T19:34:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 10 monclient(hunting): picked mon.noname-a con 0x7fb9a40a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 10 monclient(hunting): picked mon.noname-c con 0x7fb9a413f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 10 monclient(hunting): picked mon.noname-b con 0x7fb9a41430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a41daa20 con 0x7fb9a40a98f0 2024-03-25T19:34:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a41371e0 con 0x7fb9a41430c0 2024-03-25T19:34:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a41374c0 con 0x7fb9a413f710 2024-03-25T19:34:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:32.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.088+0000 7fb9a8c45640 10 monclient(hunting): authenticate will time out at 6320.269791s 2024-03-25T19:34:32.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.098+0000 7fb9a2d9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fb9a41430c0 0x7fb9a41e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:48894/0 (socket says 172.21.15.52:48894) 2024-03-25T19:34:32.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.098+0000 7fb9a2d9d640 1 -- 172.21.15.52:0/380846608 learned_addr learned my addr 172.21.15.52:0/380846608 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:32.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.100+0000 7fb99b7fe640 10 client.?.objecter ms_handle_connect 0x7fb9a41430c0 2024-03-25T19:34:32.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.100+0000 7fb99b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:32.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.102+0000 7fb99b7fe640 10 client.?.objecter ms_handle_connect 0x7fb9a413f710 2024-03-25T19:34:32.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.102+0000 7fb99b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:32.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.105+0000 7fb99b7fe640 10 client.?.objecter ms_handle_connect 0x7fb9a40a98f0 2024-03-25T19:34:32.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.105+0000 7fb99b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:32.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.111+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823036198 0 0) 0x7fb9a41374c0 con 0x7fb9a413f710 2024-03-25T19:34:32.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.111+0000 7fb99b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.111+0000 7fb99b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.111+0000 7fb99b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.111+0000 7fb99b7fe640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:34:32.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.111+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb984004eb0 con 0x7fb9a413f710 2024-03-25T19:34:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.115+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659005045 0 0) 0x7fb9a41371e0 con 0x7fb9a41430c0 2024-03-25T19:34:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.115+0000 7fb99b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.115+0000 7fb99b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.115+0000 7fb99b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.115+0000 7fb99b7fe640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:34:32.116 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.115+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a41374c0 con 0x7fb9a41430c0 2024-03-25T19:34:32.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.120+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416926136 0 0) 0x7fb9a41daa20 con 0x7fb9a40a98f0 2024-03-25T19:34:32.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.120+0000 7fb99b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:32.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.120+0000 7fb99b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:32.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.120+0000 7fb99b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:32.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.120+0000 7fb99b7fe640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:34:32.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.120+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a41371e0 con 0x7fb9a40a98f0 2024-03-25T19:34:32.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.127+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415221206 0 0) 0x7fb984004eb0 con 0x7fb9a413f710 2024-03-25T19:34:32.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.127+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9a41daa20 con 0x7fb9a413f710 2024-03-25T19:34:32.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.127+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb98c001930 con 0x7fb9a413f710 2024-03-25T19:34:32.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.127+0000 7fb99b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.127+0000 7fb99b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:32.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.127+0000 7fb99b7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:32.128 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:32.128 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:32.128 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:32.129 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:32.129 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.129 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:32.129 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:32.129 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:32.129 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:32.129 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:32.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.127+0000 7fb99b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:32.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.136+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3842751577 0 0) 0x7fb9a41371e0 con 0x7fb9a40a98f0 2024-03-25T19:34:32.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.136+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb984004eb0 con 0x7fb9a40a98f0 2024-03-25T19:34:32.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.136+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb994001ae0 con 0x7fb9a40a98f0 2024-03-25T19:34:32.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.136+0000 7fb99b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.136+0000 7fb99b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:32.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.136+0000 7fb99b7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:32.137 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:32.137 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:32.137 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:32.137 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:32.137 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.137 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:32.137 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:32.137 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:32.137 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:32.137 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:32.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3420600728 0 0) 0x7fb9a41daa20 con 0x7fb9a413f710 2024-03-25T19:34:32.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 >> v1:172.21.15.179:6789/0 conn(0x7fb9a41430c0 legacy=0x7fb9a41e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 >> v1:172.21.15.52:6789/0 conn(0x7fb9a40a98f0 legacy=0x7fb9a41352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb99b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb99b7fe640 1 monclient: found mon.c 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb99b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb99b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a413c040 con 0x7fb9a413f710 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb99b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb99b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb99b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb9a8c45640 5 monclient: authenticate success, global_id 4538 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb9a8c45640 10 monclient: _renew_subs 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb9a8c45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb9a8c45640 1 -- 172.21.15.52:0/380846608 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a41359f0 con 0x7fb9a413f710 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb9a8c45640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb9a8c45640 10 monclient: _renew_subs 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb9a8c45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb9a8c45640 1 -- 172.21.15.52:0/380846608 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a41d75d0 con 0x7fb9a413f710 2024-03-25T19:34:32.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.139+0000 7fb9a8c45640 1 librados: init done 2024-03-25T19:34:32.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.140+0000 7fb9a8c45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:32.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.140+0000 7fb9a8c45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:32.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.140+0000 7fb9a8c45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:32.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.140+0000 7fb9a8c45640 1 -- 172.21.15.52:0/380846608 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a40627b0 con 0x7fb9a413f710 2024-03-25T19:34:32.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.462+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb98c0035f0 con 0x7fb9a413f710 2024-03-25T19:34:32.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.462+0000 7fb99b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:32.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.462+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb98c003900 con 0x7fb9a413f710 2024-03-25T19:34:32.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.462+0000 7fb99b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:32.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.462+0000 7fb99b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:32.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.462+0000 7fb99b7fe640 10 monclient: dump: 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.462+0000 7fb9a159a640 4 set_mon_vals no callback set 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.463+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fb98c019910 con 0x7fb9a413f710 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.463+0000 7fb9a259c640 10 monclient: get_auth_request con 0x7fb98406dd70 auth_method 0 2024-03-25T19:34:32.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.464+0000 7fb99b7fe640 10 client.4538.objecter ms_handle_connect 0x7fb98406dd70 2024-03-25T19:34:32.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.468+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fb98c0535f0 con 0x7fb9a413f710 2024-03-25T19:34:32.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.468+0000 7fb99b7fe640 10 client.4538.objecter ms_dispatch 0x7fb9a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:32.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.468+0000 7fb99b7fe640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:32.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.468+0000 7fb99b7fe640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:32.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.469+0000 7fb99b7fe640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-25T19:34:32.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.469+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb98c04ef30 con 0x7fb9a413f710 2024-03-25T19:34:32.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.469+0000 7fb99b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:32.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.469+0000 7fb99b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:32.579 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:07:26.177 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.584 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:07:26.183 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.639 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:07:26.238 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.659 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:07:26.201 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.662 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:07:26.204 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.670 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:07:26.269 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.677 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:07:26.275 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.682 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:07:26.224 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.686+0000 7fb9a8c45640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:32.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.686+0000 7fb9a8c45640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:32.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.686+0000 7fb9a8c45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:32.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.686+0000 7fb9a8c45640 1 -- 172.21.15.52:0/380846608 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9a414f240 con 0x7fb9a413f710 2024-03-25T19:34:32.703 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:07:26.302 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:32.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.892+0000 7fb99b7fe640 1 -- 172.21.15.52:0/380846608 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fb98c051030 con 0x7fb9a413f710 2024-03-25T19:34:32.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.892+0000 7fb99b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:32.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.892+0000 7fb99b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:32.893 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:34:32.894 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:32.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.895+0000 7fb9a8c45640 10 librados: watch_flush enter 2024-03-25T19:34:32.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.895+0000 7fb9a8c45640 10 librados: watch_flush exit 2024-03-25T19:34:32.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.895+0000 7fb9a8c45640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-25T19:34:32.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.895+0000 7fb9a8c45640 10 client.4538.objecter successfully canceled tick 2024-03-25T19:34:32.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.895+0000 7fb9a8c45640 1 -- 172.21.15.52:0/380846608 >> v1:172.21.15.52:6800/3967427147 conn(0x7fb98406dd70 legacy=0x7fb984070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.895+0000 7fb9a8c45640 10 monclient: shutdown 2024-03-25T19:34:32.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.895+0000 7fb9a8c45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:32.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.895+0000 7fb9a8c45640 1 -- 172.21.15.52:0/380846608 >> v1:172.21.15.52:6790/0 conn(0x7fb9a413f710 legacy=0x7fb9a41daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:32.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.895+0000 7fb9a8c45640 1 -- 172.21.15.52:0/380846608 shutdown_connections 2024-03-25T19:34:32.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.895+0000 7fb9a8c45640 1 -- 172.21.15.52:0/380846608 >> 172.21.15.52:0/380846608 conn(0x7fb9a40b7ea0 msgr2=0x7fb9a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:32.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.896+0000 7fb9a8c45640 1 -- 172.21.15.52:0/380846608 shutdown_connections 2024-03-25T19:34:32.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.896+0000 7fb9a8c45640 1 -- 172.21.15.52:0/380846608 wait complete. 2024-03-25T19:34:32.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:32.896+0000 7fb9a8c45640 1 librados: shutdown 2024-03-25T19:34:32.907 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:39.713 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:39.836 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.836+0000 7f146519d640 10 monclient: get_monmap_and_config 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.836+0000 7f146519d640 10 monclient: build_initial_monmap 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.836+0000 7f146519d640 10 monclient: monmap: 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:34:39.837994+0000 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:34:39.837994+0000 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:34:39.837 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 1 Processor -- start 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 1 -- start start 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 10 monclient: init 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 10 monclient(hunting): picked mon.noname-c con 0x7f14600b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 10 monclient(hunting): picked mon.noname-a con 0x7f14600b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 10 monclient(hunting): picked mon.noname-b con 0x7f146014be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14600af940 con 0x7f14600b1730 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14600b0a70 con 0x7f14600b4d60 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14600b0d50 con 0x7f146014be90 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:39.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.837+0000 7f146519d640 10 monclient(hunting): authenticate will time out at 6328.018839s 2024-03-25T19:34:39.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.839+0000 7f145f59e640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f146014be90 0x7f146014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:58180/0 (socket says 172.21.15.52:58180) 2024-03-25T19:34:39.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.839+0000 7f145f59e640 1 -- 172.21.15.52:0/3084793621 learned_addr learned my addr 172.21.15.52:0/3084793621 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:39.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.850+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583320670 0 0) 0x7f14600b0a70 con 0x7f14600b4d60 2024-03-25T19:34:39.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.850+0000 7f145dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:39.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.850+0000 7f145dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:39.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.850+0000 7f145dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:39.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.850+0000 7f145dd9b640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:34:39.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.850+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1448003520 con 0x7f14600b4d60 2024-03-25T19:34:39.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.856+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147727739 0 0) 0x7f14600b0d50 con 0x7f146014be90 2024-03-25T19:34:39.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.856+0000 7f145dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:39.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.856+0000 7f145dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:39.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.856+0000 7f145dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:39.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.856+0000 7f145dd9b640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:34:39.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.856+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14600b0a70 con 0x7f146014be90 2024-03-25T19:34:39.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.871+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644657700 0 0) 0x7f14600af940 con 0x7f14600b1730 2024-03-25T19:34:39.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.871+0000 7f145dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:34:39.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.871+0000 7f145dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:39.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.871+0000 7f145dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:39.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.871+0000 7f145dd9b640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:34:39.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.871+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14600b0d50 con 0x7f14600b1730 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.877+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2810985984 0 0) 0x7f1448003520 con 0x7f14600b4d60 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.877+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 >> v1:172.21.15.52:6790/0 conn(0x7f146014be90 legacy=0x7f146014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.877+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 >> v1:172.21.15.52:6789/0 conn(0x7f14600b1730 legacy=0x7f14600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.877+0000 7f145dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.877+0000 7f145dd9b640 1 monclient: found mon.noname-c 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.877+0000 7f145dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.877+0000 7f145dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.877+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14600b1030 con 0x7f14600b4d60 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.878+0000 7f145dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.878+0000 7f145dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.878+0000 7f145dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.878+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1450001690 con 0x7f14600b4d60 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.878+0000 7f145dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:39.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.878+0000 7f145dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.878+0000 7f145dd9b640 10 monclient: dump: 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.878+0000 7f146519d640 5 monclient: authenticate success, global_id 4555 2024-03-25T19:34:39.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.878+0000 7f146519d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:39.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.889+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14500017f0 con 0x7f14600b4d60 2024-03-25T19:34:39.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.889+0000 7f145dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:39.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.889+0000 7f145dd9b640 1 -- 172.21.15.52:0/3084793621 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1450001c00 con 0x7f14600b4d60 2024-03-25T19:34:39.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.889+0000 7f146519d640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:39.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.889+0000 7f146519d640 4 set_mon_vals no callback set 2024-03-25T19:34:39.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.889+0000 7f145dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.890+0000 7f145dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.890+0000 7f145dd9b640 10 monclient: dump: 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:39.891 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:39.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.890+0000 7f146519d640 10 monclient: shutdown 2024-03-25T19:34:39.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.890+0000 7f146519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.890+0000 7f146519d640 1 -- 172.21.15.52:0/3084793621 >> v1:172.21.15.179:6789/0 conn(0x7f14600b4d60 legacy=0x7f146013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.890+0000 7f146519d640 1 -- 172.21.15.52:0/3084793621 shutdown_connections 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.891+0000 7f146519d640 1 -- 172.21.15.52:0/3084793621 >> 172.21.15.52:0/3084793621 conn(0x7f14600617a0 msgr2=0x7f1460061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.891+0000 7f146519d640 1 -- 172.21.15.52:0/3084793621 shutdown_connections 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.891+0000 7f146519d640 1 -- 172.21.15.52:0/3084793621 wait complete. 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.892+0000 7f146519d640 10 monclient: build_initial_monmap 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.892+0000 7f146519d640 10 monclient: monmap: 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:39.893 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:34:39.894 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:39.894 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:34:39.894 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:34:39.894 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:39.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.892+0000 7f146519d640 1 librados: starting msgr at 2024-03-25T19:34:39.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.892+0000 7f146519d640 1 librados: starting objecter 2024-03-25T19:34:39.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.892+0000 7f146519d640 1 Processor -- start 2024-03-25T19:34:39.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.892+0000 7f146519d640 1 -- start start 2024-03-25T19:34:39.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.892+0000 7f146519d640 1 librados: setting wanted keys 2024-03-25T19:34:39.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.892+0000 7f146519d640 1 librados: calling monclient init 2024-03-25T19:34:39.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.892+0000 7f146519d640 10 monclient: init 2024-03-25T19:34:39.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.893+0000 7f146519d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:39.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.893+0000 7f146519d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:39.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.893+0000 7f146519d640 10 monclient(hunting): picked mon.noname-c con 0x7f14600b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:39.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.893+0000 7f146519d640 10 monclient(hunting): picked mon.noname-a con 0x7f14600b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:39.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.893+0000 7f146519d640 10 monclient(hunting): picked mon.noname-b con 0x7f146014be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:39.896 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.893+0000 7f146519d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1460139530 con 0x7f14600b4d60 2024-03-25T19:34:39.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.894+0000 7f146519d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146013a820 con 0x7f146014be90 2024-03-25T19:34:39.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.894+0000 7f146519d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146013bf20 con 0x7f14600b1730 2024-03-25T19:34:39.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.894+0000 7f146519d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:39.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.894+0000 7f146519d640 10 monclient(hunting): authenticate will time out at 6328.075018s 2024-03-25T19:34:39.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.899+0000 7f145ed9d640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f14600b1730 0x7f146013b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:58234/0 (socket says 172.21.15.52:58234) 2024-03-25T19:34:39.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.899+0000 7f145ed9d640 1 -- 172.21.15.52:0/836826230 learned_addr learned my addr 172.21.15.52:0/836826230 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:39.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.900+0000 7f14477fe640 10 client.?.objecter ms_handle_connect 0x7f14600b1730 2024-03-25T19:34:39.901 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.900+0000 7f14477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:39.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.907+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197817771 0 0) 0x7f146013bf20 con 0x7f14600b1730 2024-03-25T19:34:39.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.907+0000 7f14477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:39.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.907+0000 7f14477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:39.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.907+0000 7f14477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:39.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.907+0000 7f14477fe640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:34:39.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.907+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c004eb0 con 0x7f14600b1730 2024-03-25T19:34:39.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.907+0000 7f14477fe640 10 client.?.objecter ms_handle_connect 0x7f14600b4d60 2024-03-25T19:34:39.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.907+0000 7f14477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:39.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.912+0000 7f14477fe640 10 client.?.objecter ms_handle_connect 0x7f146014be90 2024-03-25T19:34:39.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.912+0000 7f14477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:39.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.913+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257495487 0 0) 0x7f1460139530 con 0x7f14600b4d60 2024-03-25T19:34:39.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.913+0000 7f14477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:39.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.913+0000 7f14477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:39.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.913+0000 7f14477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:39.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.913+0000 7f14477fe640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:34:39.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.913+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f146013bf20 con 0x7f14600b4d60 2024-03-25T19:34:39.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.919+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 694595160 0 0) 0x7f146013bf20 con 0x7f14600b4d60 2024-03-25T19:34:39.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.919+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1460139530 con 0x7f14600b4d60 2024-03-25T19:34:39.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.919+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f14540018f0 con 0x7f14600b4d60 2024-03-25T19:34:39.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.919+0000 7f14477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:39.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.919+0000 7f14477fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:39.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.919+0000 7f14477fe640 10 monclient(hunting): dump: 2024-03-25T19:34:39.920 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:39.920 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:39.920 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:39.921 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:39.921 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:39.921 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:39.921 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:39.921 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:39.921 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:39.921 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:39.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.919+0000 7f14477fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:39.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 947704338 0 0) 0x7f1460139530 con 0x7f14600b4d60 2024-03-25T19:34:39.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 >> v1:172.21.15.52:6790/0 conn(0x7f14600b1730 legacy=0x7f146013b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:39.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 >> v1:172.21.15.179:6789/0 conn(0x7f146014be90 legacy=0x7f14601d8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:39.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f14477fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:39.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f14477fe640 1 monclient: found mon.a 2024-03-25T19:34:39.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f14477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:39.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f14477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:39.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146013c0b0 con 0x7f14600b4d60 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f14477fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f14477fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f14477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f146519d640 5 monclient: authenticate success, global_id 4551 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f146519d640 10 monclient: _renew_subs 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f146519d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f146519d640 1 -- 172.21.15.52:0/836826230 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146013af90 con 0x7f14600b4d60 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f146519d640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f146519d640 10 monclient: _renew_subs 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f146519d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f146519d640 1 -- 172.21.15.52:0/836826230 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1460143190 con 0x7f14600b4d60 2024-03-25T19:34:39.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.926+0000 7f146519d640 1 librados: init done 2024-03-25T19:34:39.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.927+0000 7f146519d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:39.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.927+0000 7f146519d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:39.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.927+0000 7f146519d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:39.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:39.927+0000 7f146519d640 1 -- 172.21.15.52:0/836826230 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14600627b0 con 0x7f14600b4d60 2024-03-25T19:34:40.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.240+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14540035f0 con 0x7f14600b4d60 2024-03-25T19:34:40.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.240+0000 7f14477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:40.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.240+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1454003900 con 0x7f14600b4d60 2024-03-25T19:34:40.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.240+0000 7f14477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:40.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.240+0000 7f14477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:40.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.240+0000 7f14477fe640 10 monclient: dump: 2024-03-25T19:34:40.241 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:40.241 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:40.241 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:40.241 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:40.241 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:40.242 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:40.242 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:40.242 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:40.242 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:40.242 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:40.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.240+0000 7f145d59a640 4 set_mon_vals no callback set 2024-03-25T19:34:40.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.241+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f14540198d0 con 0x7f14600b4d60 2024-03-25T19:34:40.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.241+0000 7f145ed9d640 10 monclient: get_auth_request con 0x7f144c06dd00 auth_method 0 2024-03-25T19:34:40.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.242+0000 7f14477fe640 10 client.4551.objecter ms_handle_connect 0x7f144c06dd00 2024-03-25T19:34:40.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.248+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f14540535d0 con 0x7f14600b4d60 2024-03-25T19:34:40.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.248+0000 7f14477fe640 10 client.4551.objecter ms_dispatch 0x7f1460000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:40.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.248+0000 7f14477fe640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:40.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.248+0000 7f14477fe640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:40.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.248+0000 7f14477fe640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-25T19:34:40.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.248+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f145404ef10 con 0x7f14600b4d60 2024-03-25T19:34:40.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.248+0000 7f14477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:40.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.248+0000 7f14477fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.469+0000 7f146519d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.469+0000 7f146519d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:40.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.469+0000 7f146519d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:34:40.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.469+0000 7f146519d640 1 -- 172.21.15.52:0/836826230 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14600b59d0 con 0x7f14600b4d60 2024-03-25T19:34:40.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.498+0000 7f14477fe640 1 -- 172.21.15.52:0/836826230 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f1454051010 con 0x7f14600b4d60 2024-03-25T19:34:40.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.498+0000 7f14477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:40.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.498+0000 7f14477fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:40.499 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:34:40.499 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.500+0000 7f146519d640 10 librados: watch_flush enter 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.500+0000 7f146519d640 10 librados: watch_flush exit 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.500+0000 7f146519d640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.500+0000 7f146519d640 10 client.4551.objecter successfully canceled tick 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.501+0000 7f146519d640 1 -- 172.21.15.52:0/836826230 >> v1:172.21.15.52:6800/3967427147 conn(0x7f144c06dd00 legacy=0x7f144c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.501+0000 7f146519d640 10 monclient: shutdown 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.501+0000 7f146519d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.501+0000 7f146519d640 1 -- 172.21.15.52:0/836826230 >> v1:172.21.15.52:6789/0 conn(0x7f14600b4d60 legacy=0x7f14601d7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.501+0000 7f146519d640 1 -- 172.21.15.52:0/836826230 shutdown_connections 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.501+0000 7f146519d640 1 -- 172.21.15.52:0/836826230 >> 172.21.15.52:0/836826230 conn(0x7f14600617a0 msgr2=0x7f14600b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.501+0000 7f146519d640 1 -- 172.21.15.52:0/836826230 shutdown_connections 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.501+0000 7f146519d640 1 -- 172.21.15.52:0/836826230 wait complete. 2024-03-25T19:34:40.502 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:40.501+0000 7f146519d640 1 librados: shutdown 2024-03-25T19:34:40.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:47.318 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:47.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.432+0000 7fb8e5100640 10 monclient: get_monmap_and_config 2024-03-25T19:34:47.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.432+0000 7fb8e5100640 10 monclient: build_initial_monmap 2024-03-25T19:34:47.433 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.432+0000 7fb8e5100640 10 monclient: monmap: 2024-03-25T19:34:47.433 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:47.433 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:47.433 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:34:47.434262+0000 2024-03-25T19:34:47.433 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:34:47.434262+0000 2024-03-25T19:34:47.434 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:47.434 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:47.434 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:47.434 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:34:47.434 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:34:47.434 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:47.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.433+0000 7fb8e5100640 1 Processor -- start 2024-03-25T19:34:47.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.433+0000 7fb8e5100640 1 -- start start 2024-03-25T19:34:47.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.433+0000 7fb8e5100640 10 monclient: init 2024-03-25T19:34:47.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.434+0000 7fb8e5100640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:47.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.434+0000 7fb8e5100640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:47.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.434+0000 7fb8e5100640 10 monclient(hunting): picked mon.noname-b con 0x7fb8e00a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:47.435 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.434+0000 7fb8e5100640 10 monclient(hunting): picked mon.noname-c con 0x7fb8e013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:47.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.434+0000 7fb8e5100640 10 monclient(hunting): picked mon.noname-a con 0x7fb8e01430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:47.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.434+0000 7fb8e5100640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e0142520 con 0x7fb8e01430c0 2024-03-25T19:34:47.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.434+0000 7fb8e5100640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e00b28a0 con 0x7fb8e013f710 2024-03-25T19:34:47.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.434+0000 7fb8e5100640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e00b2a30 con 0x7fb8e00a98f0 2024-03-25T19:34:47.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.434+0000 7fb8e5100640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:47.436 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.434+0000 7fb8e5100640 10 monclient(hunting): authenticate will time out at 6335.615399s 2024-03-25T19:34:47.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.443+0000 7fb8ded9d640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fb8e00a98f0 0x7fb8e00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37636/0 (socket says 172.21.15.52:37636) 2024-03-25T19:34:47.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.444+0000 7fb8ded9d640 1 -- 172.21.15.52:0/2153159199 learned_addr learned my addr 172.21.15.52:0/2153159199 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 1 -- 172.21.15.52:0/2153159199 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717831368 0 0) 0x7fb8e00b2a30 con 0x7fb8e00a98f0 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 10 monclient(hunting): _negotiate 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 1 -- 172.21.15.52:0/2153159199 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8cc003520 con 0x7fb8e00a98f0 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 1 -- 172.21.15.52:0/2153159199 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756591572 0 0) 0x7fb8e0142520 con 0x7fb8e01430c0 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 10 monclient(hunting): _negotiate 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:34:47.458 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.457+0000 7fb8de59c640 1 -- 172.21.15.52:0/2153159199 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e00b2a30 con 0x7fb8e01430c0 2024-03-25T19:34:47.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 1 -- 172.21.15.52:0/2153159199 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 212128462 0 0) 0x7fb8e00b2a30 con 0x7fb8e01430c0 2024-03-25T19:34:47.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 1 -- 172.21.15.52:0/2153159199 >> v1:172.21.15.52:6790/0 conn(0x7fb8e00a98f0 legacy=0x7fb8e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:47.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 1 -- 172.21.15.52:0/2153159199 >> v1:172.21.15.179:6789/0 conn(0x7fb8e013f710 legacy=0x7fb8e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:47.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 10 monclient: _finish_hunting 0 2024-03-25T19:34:47.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 1 monclient: found mon.noname-a 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 1 -- 172.21.15.52:0/2153159199 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e00b44c0 con 0x7fb8e01430c0 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 10 monclient: _finish_auth 0 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 10 monclient: _check_auth_tickets 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 1 -- 172.21.15.52:0/2153159199 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb8d80019f0 con 0x7fb8e01430c0 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8de59c640 10 monclient: dump: 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:47.472 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:47.473 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:47.473 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:47.473 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:47.473 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:47.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8e5100640 5 monclient: authenticate success, global_id 4554 2024-03-25T19:34:47.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.470+0000 7fb8e5100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:47.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.491+0000 7fb8de59c640 1 -- 172.21.15.52:0/2153159199 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8d8001b50 con 0x7fb8e01430c0 2024-03-25T19:34:47.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.491+0000 7fb8de59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:47.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.491+0000 7fb8de59c640 1 -- 172.21.15.52:0/2153159199 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb8d8001f20 con 0x7fb8e01430c0 2024-03-25T19:34:47.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.491+0000 7fb8e5100640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:47.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.491+0000 7fb8e5100640 4 set_mon_vals no callback set 2024-03-25T19:34:47.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.491+0000 7fb8de59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:47.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.491+0000 7fb8de59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:47.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.491+0000 7fb8de59c640 10 monclient: dump: 2024-03-25T19:34:47.492 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:47.492 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:47.492 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.493 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.493 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:47.493 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:47.493 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:47.493 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:47.493 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:47.493 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:47.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.492+0000 7fb8e5100640 10 monclient: shutdown 2024-03-25T19:34:47.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.492+0000 7fb8e5100640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:47.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.492+0000 7fb8e5100640 1 -- 172.21.15.52:0/2153159199 >> v1:172.21.15.52:6789/0 conn(0x7fb8e01430c0 legacy=0x7fb8e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.492+0000 7fb8e5100640 1 -- 172.21.15.52:0/2153159199 shutdown_connections 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.492+0000 7fb8e5100640 1 -- 172.21.15.52:0/2153159199 >> 172.21.15.52:0/2153159199 conn(0x7fb8e00b7ea0 msgr2=0x7fb8e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.492+0000 7fb8e5100640 1 -- 172.21.15.52:0/2153159199 shutdown_connections 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.492+0000 7fb8e5100640 1 -- 172.21.15.52:0/2153159199 wait complete. 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.493+0000 7fb8e5100640 10 monclient: build_initial_monmap 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.493+0000 7fb8e5100640 10 monclient: monmap: 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:47.495 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:34:47.496 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:34:47.496 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:47.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.494+0000 7fb8e5100640 1 librados: starting msgr at 2024-03-25T19:34:47.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.494+0000 7fb8e5100640 1 librados: starting objecter 2024-03-25T19:34:47.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.494+0000 7fb8e5100640 1 Processor -- start 2024-03-25T19:34:47.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.494+0000 7fb8e5100640 1 -- start start 2024-03-25T19:34:47.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.494+0000 7fb8e5100640 1 librados: setting wanted keys 2024-03-25T19:34:47.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.494+0000 7fb8e5100640 1 librados: calling monclient init 2024-03-25T19:34:47.496 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.494+0000 7fb8e5100640 10 monclient: init 2024-03-25T19:34:47.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.495+0000 7fb8e5100640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:47.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.495+0000 7fb8e5100640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:47.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.495+0000 7fb8e5100640 10 monclient(hunting): picked mon.noname-b con 0x7fb8e00a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:47.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.495+0000 7fb8e5100640 10 monclient(hunting): picked mon.noname-c con 0x7fb8e013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:47.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.495+0000 7fb8e5100640 10 monclient(hunting): picked mon.noname-a con 0x7fb8e01430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:47.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.495+0000 7fb8e5100640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e00b7aa0 con 0x7fb8e01430c0 2024-03-25T19:34:47.497 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.495+0000 7fb8e5100640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e013b3f0 con 0x7fb8e00a98f0 2024-03-25T19:34:47.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.495+0000 7fb8e5100640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e013b6d0 con 0x7fb8e013f710 2024-03-25T19:34:47.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.495+0000 7fb8e5100640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:47.498 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.495+0000 7fb8e5100640 10 monclient(hunting): authenticate will time out at 6335.676408s 2024-03-25T19:34:47.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.510+0000 7fb8d7fff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fb8e013f710 0x7fb8e01df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:37678/0 (socket says 172.21.15.52:37678) 2024-03-25T19:34:47.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.510+0000 7fb8d7fff640 1 -- 172.21.15.52:0/4216615997 learned_addr learned my addr 172.21.15.52:0/4216615997 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:47.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.510+0000 7fb8d77fe640 10 client.?.objecter ms_handle_connect 0x7fb8e013f710 2024-03-25T19:34:47.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.510+0000 7fb8d77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:47.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.513+0000 7fb8d77fe640 10 client.?.objecter ms_handle_connect 0x7fb8e01430c0 2024-03-25T19:34:47.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.513+0000 7fb8d77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:47.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.515+0000 7fb8d77fe640 10 client.?.objecter ms_handle_connect 0x7fb8e00a98f0 2024-03-25T19:34:47.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.515+0000 7fb8d77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:47.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.518+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873250896 0 0) 0x7fb8e013b6d0 con 0x7fb8e013f710 2024-03-25T19:34:47.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.518+0000 7fb8d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:47.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.518+0000 7fb8d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:47.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.518+0000 7fb8d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:47.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.518+0000 7fb8d77fe640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:34:47.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.518+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8c0004eb0 con 0x7fb8e013f710 2024-03-25T19:34:47.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.522+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778462779 0 0) 0x7fb8e013b3f0 con 0x7fb8e00a98f0 2024-03-25T19:34:47.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.522+0000 7fb8d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:47.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.522+0000 7fb8d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:47.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.522+0000 7fb8d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:47.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.522+0000 7fb8d77fe640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:34:47.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.522+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e013b6d0 con 0x7fb8e00a98f0 2024-03-25T19:34:47.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.527+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006866608 0 0) 0x7fb8e00b7aa0 con 0x7fb8e01430c0 2024-03-25T19:34:47.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.527+0000 7fb8d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:47.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.527+0000 7fb8d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:47.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.527+0000 7fb8d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:47.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.527+0000 7fb8d77fe640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:34:47.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.527+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e013b3f0 con 0x7fb8e01430c0 2024-03-25T19:34:47.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.538+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3391300034 0 0) 0x7fb8c0004eb0 con 0x7fb8e013f710 2024-03-25T19:34:47.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.538+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8e00b7aa0 con 0x7fb8e013f710 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.538+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb8c80019d0 con 0x7fb8e013f710 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.538+0000 7fb8d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.538+0000 7fb8d77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.538+0000 7fb8d77fe640 10 monclient(hunting): dump: 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:47.539 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.538+0000 7fb8d77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:47.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2865315544 0 0) 0x7fb8e013b3f0 con 0x7fb8e01430c0 2024-03-25T19:34:47.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8c0004eb0 con 0x7fb8e01430c0 2024-03-25T19:34:47.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb8d8001e60 con 0x7fb8e01430c0 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 10 monclient(hunting): dump: 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:47.550 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:47.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1807625229 0 0) 0x7fb8e013b6d0 con 0x7fb8e00a98f0 2024-03-25T19:34:47.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8e013b3f0 con 0x7fb8e00a98f0 2024-03-25T19:34:47.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb8d00018d0 con 0x7fb8e00a98f0 2024-03-25T19:34:47.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:47.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:47.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.549+0000 7fb8d77fe640 10 monclient(hunting): dump: 2024-03-25T19:34:47.551 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:47.551 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:47.551 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.552 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.552 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:47.552 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:47.552 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:47.552 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:47.552 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:47.552 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:47.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 116700262 0 0) 0x7fb8e00b7aa0 con 0x7fb8e013f710 2024-03-25T19:34:47.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 >> v1:172.21.15.179:6789/0 conn(0x7fb8e00a98f0 legacy=0x7fb8e0139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:47.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 >> v1:172.21.15.52:6789/0 conn(0x7fb8e01430c0 legacy=0x7fb8e01e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:47.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8d77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:47.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8d77fe640 1 monclient: found mon.c 2024-03-25T19:34:47.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:47.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8d77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:47.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e0139980 con 0x7fb8e013f710 2024-03-25T19:34:47.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8d77fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:47.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8d77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:47.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:47.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8e5100640 5 monclient: authenticate success, global_id 4550 2024-03-25T19:34:47.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8e5100640 10 monclient: _renew_subs 2024-03-25T19:34:47.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8e5100640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:47.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8e5100640 1 -- 172.21.15.52:0/4216615997 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e014e770 con 0x7fb8e013f710 2024-03-25T19:34:47.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8e5100640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:47.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8e5100640 10 monclient: _renew_subs 2024-03-25T19:34:47.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8e5100640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:47.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8e5100640 1 -- 172.21.15.52:0/4216615997 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e013baf0 con 0x7fb8e013f710 2024-03-25T19:34:47.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.552+0000 7fb8e5100640 1 librados: init done 2024-03-25T19:34:47.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.553+0000 7fb8e5100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:47.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.553+0000 7fb8e5100640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:47.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.553+0000 7fb8e5100640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:47.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.553+0000 7fb8e5100640 1 -- 172.21.15.52:0/4216615997 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e00b3ef0 con 0x7fb8e013f710 2024-03-25T19:34:47.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.879+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8c8003610 con 0x7fb8e013f710 2024-03-25T19:34:47.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.879+0000 7fb8d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:47.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.879+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fb8c8003a20 con 0x7fb8e013f710 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.879+0000 7fb8d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.879+0000 7fb8d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.879+0000 7fb8d77fe640 10 monclient: dump: 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:47.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.879+0000 7fb8ddd9b640 4 set_mon_vals no callback set 2024-03-25T19:34:47.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.880+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fb8c8003c40 con 0x7fb8e013f710 2024-03-25T19:34:47.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.880+0000 7fb8ded9d640 10 monclient: get_auth_request con 0x7fb8c006e0f0 auth_method 0 2024-03-25T19:34:47.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.880+0000 7fb8d77fe640 10 client.4550.objecter ms_handle_connect 0x7fb8c006e0f0 2024-03-25T19:34:47.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.887+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fb8c80547d0 con 0x7fb8e013f710 2024-03-25T19:34:47.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.887+0000 7fb8d77fe640 10 client.4550.objecter ms_dispatch 0x7fb8e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:47.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.887+0000 7fb8d77fe640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:47.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.887+0000 7fb8d77fe640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:47.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.887+0000 7fb8d77fe640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-25T19:34:47.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.887+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8c80502e0 con 0x7fb8e013f710 2024-03-25T19:34:47.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.887+0000 7fb8d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:47.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:47.887+0000 7fb8d77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:48.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.125+0000 7fb8e5100640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:48.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.125+0000 7fb8e5100640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:48.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.125+0000 7fb8e5100640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:34:48.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.125+0000 7fb8e5100640 1 -- 172.21.15.52:0/4216615997 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8e00b7c50 con 0x7fb8e013f710 2024-03-25T19:34:48.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.153+0000 7fb8d77fe640 1 -- 172.21.15.52:0/4216615997 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fb8c8052210 con 0x7fb8e013f710 2024-03-25T19:34:48.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.153+0000 7fb8d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:48.153 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.153+0000 7fb8d77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:48.153 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:34:48.154 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:48.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.155+0000 7fb8e5100640 10 librados: watch_flush enter 2024-03-25T19:34:48.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.155+0000 7fb8e5100640 10 librados: watch_flush exit 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.155+0000 7fb8e5100640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.155+0000 7fb8e5100640 10 client.4550.objecter successfully canceled tick 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.155+0000 7fb8e5100640 1 -- 172.21.15.52:0/4216615997 >> v1:172.21.15.52:6800/3967427147 conn(0x7fb8c006e0f0 legacy=0x7fb8c00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.155+0000 7fb8e5100640 10 monclient: shutdown 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.155+0000 7fb8e5100640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.155+0000 7fb8e5100640 1 -- 172.21.15.52:0/4216615997 >> v1:172.21.15.52:6790/0 conn(0x7fb8e013f710 legacy=0x7fb8e01df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.155+0000 7fb8e5100640 1 -- 172.21.15.52:0/4216615997 shutdown_connections 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.155+0000 7fb8e5100640 1 -- 172.21.15.52:0/4216615997 >> 172.21.15.52:0/4216615997 conn(0x7fb8e00b7ea0 msgr2=0x7fb8e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.155+0000 7fb8e5100640 1 -- 172.21.15.52:0/4216615997 shutdown_connections 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.156+0000 7fb8e5100640 1 -- 172.21.15.52:0/4216615997 wait complete. 2024-03-25T19:34:48.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:48.156+0000 7fb8e5100640 1 librados: shutdown 2024-03-25T19:34:48.169 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:52.444 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:07:46.043 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.459 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:07:46.058 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.497 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:07:46.095 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.510 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:07:46.052 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.513 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:07:46.112 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.516 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:07:46.058 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.518 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:07:46.117 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.541 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:07:46.083 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:52.549 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:07:46.148 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:54.976 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:55.090 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.090+0000 7fc6c6b4d640 10 monclient: get_monmap_and_config 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.090+0000 7fc6c6b4d640 10 monclient: build_initial_monmap 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.090+0000 7fc6c6b4d640 10 monclient: monmap: 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:34:55.091716+0000 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:34:55.091716+0000 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:34:55.091 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:55.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.090+0000 7fc6c6b4d640 1 Processor -- start 2024-03-25T19:34:55.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.091+0000 7fc6c6b4d640 1 -- start start 2024-03-25T19:34:55.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.091+0000 7fc6c6b4d640 10 monclient: init 2024-03-25T19:34:55.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.091+0000 7fc6c6b4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:55.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.091+0000 7fc6c6b4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:55.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.091+0000 7fc6c6b4d640 10 monclient(hunting): picked mon.noname-a con 0x7fc6c00a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:55.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.091+0000 7fc6c6b4d640 10 monclient(hunting): picked mon.noname-b con 0x7fc6c013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:55.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.091+0000 7fc6c6b4d640 10 monclient(hunting): picked mon.noname-c con 0x7fc6c01430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:55.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.092+0000 7fc6c6b4d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c0142520 con 0x7fc6c00a98f0 2024-03-25T19:34:55.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.092+0000 7fc6c6b4d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c00b28a0 con 0x7fc6c01430c0 2024-03-25T19:34:55.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.092+0000 7fc6c6b4d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c00b2a30 con 0x7fc6c013f710 2024-03-25T19:34:55.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.092+0000 7fc6c6b4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:55.093 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.092+0000 7fc6c6b4d640 10 monclient(hunting): authenticate will time out at 6343.272950s 2024-03-25T19:34:55.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.099+0000 7fc6c50ea640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc6c01430c0 0x7fc6c014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:55754/0 (socket says 172.21.15.52:55754) 2024-03-25T19:34:55.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.099+0000 7fc6c50ea640 1 -- 172.21.15.52:0/1782330143 learned_addr learned my addr 172.21.15.52:0/1782330143 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:55.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.105+0000 7fc6bf7fe640 1 -- 172.21.15.52:0/1782330143 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676410077 0 0) 0x7fc6c00b28a0 con 0x7fc6c01430c0 2024-03-25T19:34:55.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.105+0000 7fc6bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.105+0000 7fc6bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.105+0000 7fc6bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.105+0000 7fc6bf7fe640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:34:55.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.105+0000 7fc6bf7fe640 1 -- 172.21.15.52:0/1782330143 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6b0003520 con 0x7fc6c01430c0 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.120+0000 7fc6bf7fe640 1 -- 172.21.15.52:0/1782330143 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2848588081 0 0) 0x7fc6b0003520 con 0x7fc6c01430c0 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 1 -- 172.21.15.52:0/1782330143 >> v1:172.21.15.52:6790/0 conn(0x7fc6c013f710 legacy=0x7fc6c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 1 -- 172.21.15.52:0/1782330143 >> v1:172.21.15.52:6789/0 conn(0x7fc6c00a98f0 legacy=0x7fc6c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 1 monclient: found mon.noname-c 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 1 -- 172.21.15.52:0/1782330143 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c00b44c0 con 0x7fc6c01430c0 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:55.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 1 -- 172.21.15.52:0/1782330143 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc6b8001a10 con 0x7fc6c01430c0 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6c6b4d640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6c6b4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6bf7fe640 10 monclient: dump: 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:55.123 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.121+0000 7fc6c6b4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:55.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.131+0000 7fc6bf7fe640 1 -- 172.21.15.52:0/1782330143 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6b8001ca0 con 0x7fc6c01430c0 2024-03-25T19:34:55.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.131+0000 7fc6bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:55.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.131+0000 7fc6bf7fe640 1 -- 172.21.15.52:0/1782330143 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc6b8002030 con 0x7fc6c01430c0 2024-03-25T19:34:55.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.131+0000 7fc6bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.131+0000 7fc6bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:55.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.131+0000 7fc6bf7fe640 10 monclient: dump: 2024-03-25T19:34:55.132 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:55.132 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:55.133 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.133 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.133 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.133 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:55.133 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:55.133 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:55.133 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:55.133 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:55.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.131+0000 7fc6c6b4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:55.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.131+0000 7fc6c6b4d640 4 set_mon_vals no callback set 2024-03-25T19:34:55.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.132+0000 7fc6c6b4d640 10 monclient: shutdown 2024-03-25T19:34:55.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.132+0000 7fc6c6b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.132+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/1782330143 >> v1:172.21.15.179:6789/0 conn(0x7fc6c01430c0 legacy=0x7fc6c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.132+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/1782330143 shutdown_connections 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.132+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/1782330143 >> 172.21.15.52:0/1782330143 conn(0x7fc6c00b7ea0 msgr2=0x7fc6c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.132+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/1782330143 shutdown_connections 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.132+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/1782330143 wait complete. 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.133+0000 7fc6c6b4d640 10 monclient: build_initial_monmap 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.133+0000 7fc6c6b4d640 10 monclient: monmap: 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:34:55.135 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:34:55.136 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:34:55.136 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:55.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.133+0000 7fc6c6b4d640 1 librados: starting msgr at 2024-03-25T19:34:55.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.133+0000 7fc6c6b4d640 1 librados: starting objecter 2024-03-25T19:34:55.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.133+0000 7fc6c6b4d640 1 Processor -- start 2024-03-25T19:34:55.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.133+0000 7fc6c6b4d640 1 -- start start 2024-03-25T19:34:55.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.133+0000 7fc6c6b4d640 1 librados: setting wanted keys 2024-03-25T19:34:55.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.133+0000 7fc6c6b4d640 1 librados: calling monclient init 2024-03-25T19:34:55.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.133+0000 7fc6c6b4d640 10 monclient: init 2024-03-25T19:34:55.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.134+0000 7fc6c6b4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:55.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.134+0000 7fc6c6b4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:34:55.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.134+0000 7fc6c6b4d640 10 monclient(hunting): picked mon.noname-c con 0x7fc6c00a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:34:55.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.134+0000 7fc6c6b4d640 10 monclient(hunting): picked mon.noname-b con 0x7fc6c013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:34:55.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.135+0000 7fc6c6b4d640 10 monclient(hunting): picked mon.noname-a con 0x7fc6c01430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:34:55.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.135+0000 7fc6c6b4d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c00b1780 con 0x7fc6c01430c0 2024-03-25T19:34:55.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.135+0000 7fc6c6b4d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c01398d0 con 0x7fc6c013f710 2024-03-25T19:34:55.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.135+0000 7fc6c6b4d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c00abb00 con 0x7fc6c00a98f0 2024-03-25T19:34:55.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.135+0000 7fc6c6b4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:55.137 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.135+0000 7fc6c6b4d640 10 monclient(hunting): authenticate will time out at 6343.316009s 2024-03-25T19:34:55.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.141+0000 7fc6bffff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc6c013f710 0x7fc6c01db040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:55774/0 (socket says 172.21.15.52:55774) 2024-03-25T19:34:55.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.141+0000 7fc6bffff640 1 -- 172.21.15.52:0/818105666 learned_addr learned my addr 172.21.15.52:0/818105666 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:34:55.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.142+0000 7fc6bd7fa640 10 client.?.objecter ms_handle_connect 0x7fc6c013f710 2024-03-25T19:34:55.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.142+0000 7fc6bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:55.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.153+0000 7fc6bd7fa640 10 client.?.objecter ms_handle_connect 0x7fc6c01430c0 2024-03-25T19:34:55.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.153+0000 7fc6bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:55.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.154+0000 7fc6bd7fa640 10 client.?.objecter ms_handle_connect 0x7fc6c00a98f0 2024-03-25T19:34:55.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.154+0000 7fc6bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:55.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.163+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295848729 0 0) 0x7fc6c01398d0 con 0x7fc6c013f710 2024-03-25T19:34:55.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.163+0000 7fc6bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.163+0000 7fc6bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.163+0000 7fc6bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.163+0000 7fc6bd7fa640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:34:55.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.163+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a4004eb0 con 0x7fc6c013f710 2024-03-25T19:34:55.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.170+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407231384 0 0) 0x7fc6c00abb00 con 0x7fc6c00a98f0 2024-03-25T19:34:55.170 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.170+0000 7fc6bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.170+0000 7fc6bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.170+0000 7fc6bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.170+0000 7fc6bd7fa640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:34:55.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.170+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c01398d0 con 0x7fc6c00a98f0 2024-03-25T19:34:55.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.180+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598847950 0 0) 0x7fc6c00b1780 con 0x7fc6c01430c0 2024-03-25T19:34:55.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.180+0000 7fc6bd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:55.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.180+0000 7fc6bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:55.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.180+0000 7fc6bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:55.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.180+0000 7fc6bd7fa640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:34:55.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.180+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c00abb00 con 0x7fc6c01430c0 2024-03-25T19:34:55.189 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.189+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3711566207 0 0) 0x7fc6a4004eb0 con 0x7fc6c013f710 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.189+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6c00b1780 con 0x7fc6c013f710 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.189+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc6ac001a30 con 0x7fc6c013f710 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.189+0000 7fc6bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.189+0000 7fc6bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.189+0000 7fc6bd7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:55.190 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:55.191 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:55.191 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:55.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.189+0000 7fc6bd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:55.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.197+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4282480440 0 0) 0x7fc6c01398d0 con 0x7fc6c00a98f0 2024-03-25T19:34:55.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.197+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6a4004eb0 con 0x7fc6c00a98f0 2024-03-25T19:34:55.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.197+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc6b4001510 con 0x7fc6c00a98f0 2024-03-25T19:34:55.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.197+0000 7fc6bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.197+0000 7fc6bd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:55.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.197+0000 7fc6bd7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:55.198 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:55.198 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:55.198 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.198 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.199 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.199 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:55.199 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:55.199 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:55.199 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:55.199 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:55.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.205+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3795696141 0 0) 0x7fc6c00abb00 con 0x7fc6c01430c0 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.205+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6c01398d0 con 0x7fc6c01430c0 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.205+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc6b80018d0 con 0x7fc6c01430c0 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.205+0000 7fc6bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.205+0000 7fc6bd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.205+0000 7fc6bd7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:55.206 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:55.207 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4115559648 0 0) 0x7fc6c00b1780 con 0x7fc6c013f710 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 >> v1:172.21.15.52:6790/0 conn(0x7fc6c00a98f0 legacy=0x7fc6c0138570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 >> v1:172.21.15.52:6789/0 conn(0x7fc6c01430c0 legacy=0x7fc6c01e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6bd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6bd7fa640 1 monclient: found mon.b 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6bd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c0137960 con 0x7fc6c013f710 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6bd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6bd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:55.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6c6b4d640 5 monclient: authenticate success, global_id 4570 2024-03-25T19:34:55.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6c6b4d640 10 monclient: _renew_subs 2024-03-25T19:34:55.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6c6b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:34:55.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/818105666 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c01db750 con 0x7fc6c013f710 2024-03-25T19:34:55.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6c6b4d640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:55.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6c6b4d640 10 monclient: _renew_subs 2024-03-25T19:34:55.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6c6b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:34:55.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/818105666 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c0137e40 con 0x7fc6c013f710 2024-03-25T19:34:55.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.210+0000 7fc6c6b4d640 1 librados: init done 2024-03-25T19:34:55.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.211+0000 7fc6c6b4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:55.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.211+0000 7fc6c6b4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:55.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.211+0000 7fc6c6b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:34:55.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.211+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/818105666 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c00b3ef0 con 0x7fc6c013f710 2024-03-25T19:34:55.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.480+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6ac001c70 con 0x7fc6c013f710 2024-03-25T19:34:55.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.480+0000 7fc6bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:55.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.480+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc6ac003a60 con 0x7fc6c013f710 2024-03-25T19:34:55.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.480+0000 7fc6bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:55.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.480+0000 7fc6bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:55.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.480+0000 7fc6bd7fa640 10 monclient: dump: 2024-03-25T19:34:55.482 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:34:55.482 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:34:55.482 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.482 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:34:55.482 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:55.482 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:34:55.482 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:34:55.482 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:34:55.482 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:34:55.482 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:34:55.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.480+0000 7fc6beffd640 4 set_mon_vals no callback set 2024-03-25T19:34:55.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.481+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fc6ac003c40 con 0x7fc6c013f710 2024-03-25T19:34:55.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.482+0000 7fc6c48e9640 10 monclient: get_auth_request con 0x7fc6a406e0b0 auth_method 0 2024-03-25T19:34:55.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.482+0000 7fc6bd7fa640 10 client.4570.objecter ms_handle_connect 0x7fc6a406e0b0 2024-03-25T19:34:55.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.485+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fc6ac0548e0 con 0x7fc6c013f710 2024-03-25T19:34:55.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.485+0000 7fc6bd7fa640 10 client.4570.objecter ms_dispatch 0x7fc6c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:55.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.485+0000 7fc6bd7fa640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:55.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.485+0000 7fc6bd7fa640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:55.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.485+0000 7fc6bd7fa640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T19:34:55.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.486+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6ac0503f0 con 0x7fc6c013f710 2024-03-25T19:34:55.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.486+0000 7fc6bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:55.486 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.486+0000 7fc6bd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.739+0000 7fc6c6b4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.739+0000 7fc6c6b4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.739+0000 7fc6c6b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:34:55.740 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.739+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/818105666 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6c00627b0 con 0x7fc6c013f710 2024-03-25T19:34:55.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.786+0000 7fc6bd7fa640 1 -- 172.21.15.52:0/818105666 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fc6ac052320 con 0x7fc6c013f710 2024-03-25T19:34:55.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.786+0000 7fc6bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:55.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.786+0000 7fc6bd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:55.787 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:34:55.787 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:55.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.788+0000 7fc6c6b4d640 10 librados: watch_flush enter 2024-03-25T19:34:55.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.788+0000 7fc6c6b4d640 10 librados: watch_flush exit 2024-03-25T19:34:55.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.788+0000 7fc6c6b4d640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T19:34:55.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.788+0000 7fc6c6b4d640 10 client.4570.objecter successfully canceled tick 2024-03-25T19:34:55.789 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.788+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/818105666 >> v1:172.21.15.52:6800/3967427147 conn(0x7fc6a406e0b0 legacy=0x7fc6a4070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.788+0000 7fc6c6b4d640 10 monclient: shutdown 2024-03-25T19:34:55.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.788+0000 7fc6c6b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:55.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.788+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/818105666 >> v1:172.21.15.179:6789/0 conn(0x7fc6c013f710 legacy=0x7fc6c01db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:55.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.789+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/818105666 shutdown_connections 2024-03-25T19:34:55.790 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.789+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/818105666 >> 172.21.15.52:0/818105666 conn(0x7fc6c00b7ea0 msgr2=0x7fc6c013c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:55.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.789+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/818105666 shutdown_connections 2024-03-25T19:34:55.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.789+0000 7fc6c6b4d640 1 -- 172.21.15.52:0/818105666 wait complete. 2024-03-25T19:34:55.791 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:34:55.789+0000 7fc6c6b4d640 1 librados: shutdown 2024-03-25T19:34:55.803 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:02.609 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:02.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.726+0000 7f6884865640 10 monclient: get_monmap_and_config 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.726+0000 7f6884865640 10 monclient: build_initial_monmap 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.726+0000 7f6884865640 10 monclient: monmap: 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:35:02.727746+0000 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:35:02.727746+0000 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:35:02.727 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.726+0000 7f6884865640 1 Processor -- start 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.726+0000 7f6884865640 1 -- start start 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.726+0000 7f6884865640 10 monclient: init 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.727+0000 7f6884865640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.727+0000 7f6884865640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.727+0000 7f6884865640 10 monclient(hunting): picked mon.noname-a con 0x7f68800a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.727+0000 7f6884865640 10 monclient(hunting): picked mon.noname-c con 0x7f688013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.727+0000 7f6884865640 10 monclient(hunting): picked mon.noname-b con 0x7f68801430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.727+0000 7f6884865640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6880142520 con 0x7f68800a98f0 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.727+0000 7f6884865640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68800b28a0 con 0x7f688013f710 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.727+0000 7f6884865640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68800b2a30 con 0x7f68801430c0 2024-03-25T19:35:02.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.727+0000 7f6884865640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:02.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.727+0000 7f6884865640 10 monclient(hunting): authenticate will time out at 6350.908796s 2024-03-25T19:35:02.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.734+0000 7f687ed9d640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f68801430c0 0x7f688014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:50320/0 (socket says 172.21.15.52:50320) 2024-03-25T19:35:02.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.734+0000 7f687ed9d640 1 -- 172.21.15.52:0/2983059079 learned_addr learned my addr 172.21.15.52:0/2983059079 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:02.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.742+0000 7f687dd9b640 1 -- 172.21.15.52:0/2983059079 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969774232 0 0) 0x7f68800b2a30 con 0x7f68801430c0 2024-03-25T19:35:02.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.742+0000 7f687dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:02.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.742+0000 7f687dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:02.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.742+0000 7f687dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:02.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.742+0000 7f687dd9b640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:35:02.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.742+0000 7f687dd9b640 1 -- 172.21.15.52:0/2983059079 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686c003520 con 0x7f68801430c0 2024-03-25T19:35:02.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.748+0000 7f687dd9b640 1 -- 172.21.15.52:0/2983059079 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556536645 0 0) 0x7f68800b28a0 con 0x7f688013f710 2024-03-25T19:35:02.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.748+0000 7f687dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:02.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.748+0000 7f687dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:02.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.748+0000 7f687dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:02.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.748+0000 7f687dd9b640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:35:02.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.748+0000 7f687dd9b640 1 -- 172.21.15.52:0/2983059079 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68800b2a30 con 0x7f688013f710 2024-03-25T19:35:02.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 1 -- 172.21.15.52:0/2983059079 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1350277346 0 0) 0x7f686c003520 con 0x7f68801430c0 2024-03-25T19:35:02.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 1 -- 172.21.15.52:0/2983059079 >> v1:172.21.15.179:6789/0 conn(0x7f688013f710 legacy=0x7f68801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:02.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 1 -- 172.21.15.52:0/2983059079 >> v1:172.21.15.52:6789/0 conn(0x7f68800a98f0 legacy=0x7f68800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:35:02.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:35:02.757 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 1 monclient: found mon.noname-b 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 1 -- 172.21.15.52:0/2983059079 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68800b44c0 con 0x7f68801430c0 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 1 -- 172.21.15.52:0/2983059079 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6878001a10 con 0x7f68801430c0 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f6884865640 5 monclient: authenticate success, global_id 4559 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f6884865640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f687dd9b640 10 monclient: dump: 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:02.758 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:02.759 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:02.759 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:02.759 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:02.759 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:02.759 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:02.759 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:02.759 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:02.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.756+0000 7f6884865640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:02.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.769+0000 7f687dd9b640 1 -- 172.21.15.52:0/2983059079 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6878001b70 con 0x7f68801430c0 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.769+0000 7f687dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.769+0000 7f687dd9b640 1 -- 172.21.15.52:0/2983059079 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6878001f80 con 0x7f68801430c0 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.769+0000 7f687dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.769+0000 7f687dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.769+0000 7f687dd9b640 10 monclient: dump: 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:02.771 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:02.772 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:02.772 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:02.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.769+0000 7f6884865640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:02.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.769+0000 7f6884865640 4 set_mon_vals no callback set 2024-03-25T19:35:02.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.770+0000 7f6884865640 10 monclient: shutdown 2024-03-25T19:35:02.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.770+0000 7f6884865640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:02.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.770+0000 7f6884865640 1 -- 172.21.15.52:0/2983059079 >> v1:172.21.15.52:6790/0 conn(0x7f68801430c0 legacy=0x7f688014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:02.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.771+0000 7f6884865640 1 -- 172.21.15.52:0/2983059079 shutdown_connections 2024-03-25T19:35:02.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.771+0000 7f6884865640 1 -- 172.21.15.52:0/2983059079 >> 172.21.15.52:0/2983059079 conn(0x7f68800b7ea0 msgr2=0x7f68800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:02.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.771+0000 7f6884865640 1 -- 172.21.15.52:0/2983059079 shutdown_connections 2024-03-25T19:35:02.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.771+0000 7f6884865640 1 -- 172.21.15.52:0/2983059079 wait complete. 2024-03-25T19:35:02.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.772+0000 7f6884865640 10 monclient: build_initial_monmap 2024-03-25T19:35:02.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.772+0000 7f6884865640 10 monclient: monmap: 2024-03-25T19:35:02.773 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:02.773 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:02.773 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:35:02.774 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:35:02.774 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:02.774 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:35:02.774 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:02.774 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:35:02.774 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:35:02.774 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:02.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.772+0000 7f6884865640 1 librados: starting msgr at 2024-03-25T19:35:02.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.772+0000 7f6884865640 1 librados: starting objecter 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.773+0000 7f6884865640 1 Processor -- start 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.773+0000 7f6884865640 1 -- start start 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.773+0000 7f6884865640 1 librados: setting wanted keys 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.773+0000 7f6884865640 1 librados: calling monclient init 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.773+0000 7f6884865640 10 monclient: init 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.774+0000 7f6884865640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.774+0000 7f6884865640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.774+0000 7f6884865640 10 monclient(hunting): picked mon.noname-c con 0x7f68800a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.774+0000 7f6884865640 10 monclient(hunting): picked mon.noname-b con 0x7f688013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.774+0000 7f6884865640 10 monclient(hunting): picked mon.noname-a con 0x7f68801430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.774+0000 7f6884865640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68801373f0 con 0x7f68801430c0 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.774+0000 7f6884865640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68801db280 con 0x7f688013f710 2024-03-25T19:35:02.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.774+0000 7f6884865640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6880138f40 con 0x7f68800a98f0 2024-03-25T19:35:02.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.774+0000 7f6884865640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:02.777 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.774+0000 7f6884865640 10 monclient(hunting): authenticate will time out at 6350.955312s 2024-03-25T19:35:02.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.780+0000 7f687e59c640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f68800a98f0 0x7f68801dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:50360/0 (socket says 172.21.15.52:50360) 2024-03-25T19:35:02.781 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.780+0000 7f687e59c640 1 -- 172.21.15.52:0/2058493772 learned_addr learned my addr 172.21.15.52:0/2058493772 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:02.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.782+0000 7f68777fe640 10 client.?.objecter ms_handle_connect 0x7f68800a98f0 2024-03-25T19:35:02.782 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.782+0000 7f68777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:02.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.784+0000 7f68777fe640 10 client.?.objecter ms_handle_connect 0x7f688013f710 2024-03-25T19:35:02.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.784+0000 7f68777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:02.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.792+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318190006 0 0) 0x7f6880138f40 con 0x7f68800a98f0 2024-03-25T19:35:02.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.792+0000 7f68777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:02.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.792+0000 7f68777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:02.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.792+0000 7f68777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:02.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.792+0000 7f68777fe640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:35:02.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.792+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6860004eb0 con 0x7f68800a98f0 2024-03-25T19:35:02.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.801+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623917495 0 0) 0x7f68801db280 con 0x7f688013f710 2024-03-25T19:35:02.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.801+0000 7f68777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:02.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.801+0000 7f68777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:02.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.801+0000 7f68777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:02.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.801+0000 7f68777fe640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:35:02.802 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.801+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6880138f40 con 0x7f688013f710 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.803+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1009960499 0 0) 0x7f6860004eb0 con 0x7f68800a98f0 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.803+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68801db280 con 0x7f68800a98f0 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.803+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6870001960 con 0x7f68800a98f0 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.803+0000 7f68777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.803+0000 7f68777fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.803+0000 7f68777fe640 10 monclient(hunting): dump: 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:02.804 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:02.805 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:02.805 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:02.805 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:02.805 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.803+0000 7f68777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:02.807 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.807+0000 7f68777fe640 10 client.?.objecter ms_handle_connect 0x7f68801430c0 2024-03-25T19:35:02.808 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.807+0000 7f68777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:02.818 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 405834544 0 0) 0x7f68801db280 con 0x7f68800a98f0 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 >> v1:172.21.15.179:6789/0 conn(0x7f688013f710 legacy=0x7f6880132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 >> v1:172.21.15.52:6789/0 conn(0x7f68801430c0 legacy=0x7f68801332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f68777fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f68777fe640 1 monclient: found mon.c 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f68777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f68777fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68801339d0 con 0x7f68800a98f0 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f68777fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f68777fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f68777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f6884865640 5 monclient: authenticate success, global_id 4562 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f6884865640 10 monclient: _renew_subs 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f6884865640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.817+0000 7f6884865640 1 -- 172.21.15.52:0/2058493772 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6880137f50 con 0x7f68800a98f0 2024-03-25T19:35:02.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.818+0000 7f6884865640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:02.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.818+0000 7f6884865640 10 monclient: _renew_subs 2024-03-25T19:35:02.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.818+0000 7f6884865640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:02.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.818+0000 7f6884865640 1 -- 172.21.15.52:0/2058493772 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f688013c550 con 0x7f68800a98f0 2024-03-25T19:35:02.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.818+0000 7f6884865640 1 librados: init done 2024-03-25T19:35:02.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.819+0000 7f6884865640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:02.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.819+0000 7f6884865640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:02.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.819+0000 7f6884865640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:02.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:02.819+0000 7f6884865640 1 -- 172.21.15.52:0/2058493772 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68800b3ef0 con 0x7f68800a98f0 2024-03-25T19:35:03.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.138+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68700035f0 con 0x7f68800a98f0 2024-03-25T19:35:03.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.138+0000 7f68777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:03.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.138+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6870003940 con 0x7f68800a98f0 2024-03-25T19:35:03.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.138+0000 7f68777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:03.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.138+0000 7f68777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:03.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.138+0000 7f68777fe640 10 monclient: dump: 2024-03-25T19:35:03.139 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:03.140 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:03.140 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:03.140 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:03.140 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:03.140 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:03.140 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:03.140 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:03.140 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:03.140 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:03.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.138+0000 7f687d59a640 4 set_mon_vals no callback set 2024-03-25T19:35:03.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.139+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f6870003b60 con 0x7f68800a98f0 2024-03-25T19:35:03.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.140+0000 7f6875d9b640 10 monclient: get_auth_request con 0x7f686006dd10 auth_method 0 2024-03-25T19:35:03.141 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.140+0000 7f68777fe640 10 client.4562.objecter ms_handle_connect 0x7f686006dd10 2024-03-25T19:35:03.147 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.147+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f68700547f0 con 0x7f68800a98f0 2024-03-25T19:35:03.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.147+0000 7f68777fe640 10 client.4562.objecter ms_dispatch 0x7f6880000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:03.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.147+0000 7f68777fe640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:03.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.147+0000 7f68777fe640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:03.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.147+0000 7f68777fe640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-25T19:35:03.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.147+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6870050300 con 0x7f68800a98f0 2024-03-25T19:35:03.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.147+0000 7f68777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:03.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.147+0000 7f68777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:03.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.393+0000 7f6884865640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:03.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.393+0000 7f6884865640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:03.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.393+0000 7f6884865640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:03.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.393+0000 7f6884865640 1 -- 172.21.15.52:0/2058493772 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68800627b0 con 0x7f68800a98f0 2024-03-25T19:35:03.420 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.420+0000 7f68777fe640 1 -- 172.21.15.52:0/2058493772 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f6870052230 con 0x7f68800a98f0 2024-03-25T19:35:03.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.420+0000 7f68777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:03.421 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.420+0000 7f68777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:03.421 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:35:03.421 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 10 librados: watch_flush enter 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 10 librados: watch_flush exit 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 10 client.4562.objecter successfully canceled tick 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 1 -- 172.21.15.52:0/2058493772 >> v1:172.21.15.52:6800/3967427147 conn(0x7f686006dd10 legacy=0x7f68600701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 10 monclient: shutdown 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 1 -- 172.21.15.52:0/2058493772 >> v1:172.21.15.52:6790/0 conn(0x7f68800a98f0 legacy=0x7f68801dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 1 -- 172.21.15.52:0/2058493772 shutdown_connections 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 1 -- 172.21.15.52:0/2058493772 >> 172.21.15.52:0/2058493772 conn(0x7f68800b7ea0 msgr2=0x7f6880074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:03.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 1 -- 172.21.15.52:0/2058493772 shutdown_connections 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 1 -- 172.21.15.52:0/2058493772 wait complete. 2024-03-25T19:35:03.424 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:03.422+0000 7f6884865640 1 librados: shutdown 2024-03-25T19:35:03.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:10.240 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:10.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.354+0000 7f6cb4a93640 10 monclient: get_monmap_and_config 2024-03-25T19:35:10.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.354+0000 7f6cb4a93640 10 monclient: build_initial_monmap 2024-03-25T19:35:10.354 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.354+0000 7f6cb4a93640 10 monclient: monmap: 2024-03-25T19:35:10.354 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:10.354 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:10.354 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:35:10.355291+0000 2024-03-25T19:35:10.354 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:35:10.355291+0000 2024-03-25T19:35:10.354 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:10.354 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:10.354 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:10.355 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:35:10.355 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:35:10.355 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:10.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.355+0000 7f6cb4a93640 1 Processor -- start 2024-03-25T19:35:10.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.355+0000 7f6cb4a93640 1 -- start start 2024-03-25T19:35:10.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.355+0000 7f6cb4a93640 10 monclient: init 2024-03-25T19:35:10.355 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.356+0000 7f6cb4a93640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:10.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.356+0000 7f6cb4a93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:10.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.356+0000 7f6cb4a93640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb00a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:10.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.356+0000 7f6cb4a93640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:10.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.356+0000 7f6cb4a93640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb01430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:10.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.356+0000 7f6cb4a93640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb0142520 con 0x7f6cb00a98f0 2024-03-25T19:35:10.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.356+0000 7f6cb4a93640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb00b28a0 con 0x7f6cb01430c0 2024-03-25T19:35:10.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.356+0000 7f6cb4a93640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb00b2a30 con 0x7f6cb013f710 2024-03-25T19:35:10.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.356+0000 7f6cb4a93640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:10.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.356+0000 7f6cb4a93640 10 monclient(hunting): authenticate will time out at 6358.536329s 2024-03-25T19:35:10.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.365+0000 7f6cae59c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f6cb00a98f0 0x7f6cb00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:49684/0 (socket says 172.21.15.52:49684) 2024-03-25T19:35:10.365 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.365+0000 7f6cae59c640 1 -- 172.21.15.52:0/1126496426 learned_addr learned my addr 172.21.15.52:0/1126496426 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:10.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.379+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201683239 0 0) 0x7f6cb00b2a30 con 0x7f6cb013f710 2024-03-25T19:35:10.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.379+0000 7f6cad59a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:10.378 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.379+0000 7f6cad59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:10.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.379+0000 7f6cad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:10.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.379+0000 7f6cad59a640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:35:10.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.379+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c98003520 con 0x7f6cb013f710 2024-03-25T19:35:10.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.386+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345001808 0 0) 0x7f6cb0142520 con 0x7f6cb00a98f0 2024-03-25T19:35:10.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.386+0000 7f6cad59a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:10.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.386+0000 7f6cad59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:10.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.386+0000 7f6cad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:10.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.386+0000 7f6cad59a640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:35:10.386 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.386+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb00b2a30 con 0x7f6cb00a98f0 2024-03-25T19:35:10.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.394+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237824388 0 0) 0x7f6cb00b28a0 con 0x7f6cb01430c0 2024-03-25T19:35:10.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.394+0000 7f6cad59a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:10.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.394+0000 7f6cad59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:10.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.394+0000 7f6cad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:10.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.394+0000 7f6cad59a640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:35:10.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.394+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c980039d0 con 0x7f6cb01430c0 2024-03-25T19:35:10.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1613242334 0 0) 0x7f6c98003520 con 0x7f6cb013f710 2024-03-25T19:35:10.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 >> v1:172.21.15.179:6789/0 conn(0x7f6cb01430c0 legacy=0x7f6cb014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:10.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 >> v1:172.21.15.52:6789/0 conn(0x7f6cb00a98f0 legacy=0x7f6cb00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:10.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 10 monclient: _finish_hunting 0 2024-03-25T19:35:10.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 1 monclient: found mon.noname-b 2024-03-25T19:35:10.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:10.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:35:10.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb00b44c0 con 0x7f6cb013f710 2024-03-25T19:35:10.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 10 monclient: _finish_auth 0 2024-03-25T19:35:10.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 10 monclient: _check_auth_tickets 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6ca4001690 con 0x7f6cb013f710 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cad59a640 10 monclient: dump: 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cb4a93640 5 monclient: authenticate success, global_id 4565 2024-03-25T19:35:10.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.398+0000 7f6cb4a93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:10.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.410+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ca40017f0 con 0x7f6cb013f710 2024-03-25T19:35:10.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.410+0000 7f6cad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:10.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.410+0000 7f6cad59a640 1 -- 172.21.15.52:0/1126496426 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6ca4001c00 con 0x7f6cb013f710 2024-03-25T19:35:10.409 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.410+0000 7f6cb4a93640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.410+0000 7f6cb4a93640 4 set_mon_vals no callback set 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.410+0000 7f6cad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.410+0000 7f6cad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.410+0000 7f6cad59a640 10 monclient: dump: 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:10.410 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:10.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.411+0000 7f6cb4a93640 10 monclient: shutdown 2024-03-25T19:35:10.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.411+0000 7f6cb4a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:10.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.411+0000 7f6cb4a93640 1 -- 172.21.15.52:0/1126496426 >> v1:172.21.15.52:6790/0 conn(0x7f6cb013f710 legacy=0x7f6cb01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:10.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.411+0000 7f6cb4a93640 1 -- 172.21.15.52:0/1126496426 shutdown_connections 2024-03-25T19:35:10.411 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.411+0000 7f6cb4a93640 1 -- 172.21.15.52:0/1126496426 >> 172.21.15.52:0/1126496426 conn(0x7f6cb00b7ea0 msgr2=0x7f6cb00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:10.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.411+0000 7f6cb4a93640 1 -- 172.21.15.52:0/1126496426 shutdown_connections 2024-03-25T19:35:10.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.411+0000 7f6cb4a93640 1 -- 172.21.15.52:0/1126496426 wait complete. 2024-03-25T19:35:10.412 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.412+0000 7f6cb4a93640 10 monclient: build_initial_monmap 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.412+0000 7f6cb4a93640 10 monclient: monmap: 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.413+0000 7f6cb4a93640 1 librados: starting msgr at 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.413+0000 7f6cb4a93640 1 librados: starting objecter 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.413+0000 7f6cb4a93640 1 Processor -- start 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.413+0000 7f6cb4a93640 1 -- start start 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.413+0000 7f6cb4a93640 1 librados: setting wanted keys 2024-03-25T19:35:10.413 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.413+0000 7f6cb4a93640 1 librados: calling monclient init 2024-03-25T19:35:10.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.413+0000 7f6cb4a93640 10 monclient: init 2024-03-25T19:35:10.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.414+0000 7f6cb4a93640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:10.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.414+0000 7f6cb4a93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:10.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.414+0000 7f6cb4a93640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb00a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:10.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.414+0000 7f6cb4a93640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb013f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:10.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.414+0000 7f6cb4a93640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb01430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:10.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.414+0000 7f6cb4a93640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb0133ad0 con 0x7f6cb00a98f0 2024-03-25T19:35:10.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.414+0000 7f6cb4a93640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb00abe70 con 0x7f6cb01430c0 2024-03-25T19:35:10.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.414+0000 7f6cb4a93640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb00ac150 con 0x7f6cb013f710 2024-03-25T19:35:10.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.415+0000 7f6cb4a93640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:10.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.415+0000 7f6cb4a93640 10 monclient(hunting): authenticate will time out at 6358.595025s 2024-03-25T19:35:10.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.424+0000 7f6caed9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f6cb01430c0 0x7f6cb01333c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:50116/0 (socket says 172.21.15.52:50116) 2024-03-25T19:35:10.423 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.424+0000 7f6caed9d640 1 -- 172.21.15.52:0/2373681232 learned_addr learned my addr 172.21.15.52:0/2373681232 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:10.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.426+0000 7f6c9effd640 10 client.?.objecter ms_handle_connect 0x7f6cb01430c0 2024-03-25T19:35:10.426 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.426+0000 7f6c9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:10.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.439+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916197825 0 0) 0x7f6cb00abe70 con 0x7f6cb01430c0 2024-03-25T19:35:10.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.439+0000 7f6c9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:10.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.439+0000 7f6c9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:10.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.439+0000 7f6c9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:10.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.439+0000 7f6c9effd640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:35:10.440 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.439+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c90004eb0 con 0x7f6cb01430c0 2024-03-25T19:35:10.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.444+0000 7f6c9effd640 10 client.?.objecter ms_handle_connect 0x7f6cb013f710 2024-03-25T19:35:10.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.444+0000 7f6c9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:10.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.454+0000 7f6c9effd640 10 client.?.objecter ms_handle_connect 0x7f6cb00a98f0 2024-03-25T19:35:10.453 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.454+0000 7f6c9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:10.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.460+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007867613 0 0) 0x7f6cb0133ad0 con 0x7f6cb00a98f0 2024-03-25T19:35:10.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.460+0000 7f6c9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:10.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.460+0000 7f6c9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:10.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.460+0000 7f6c9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:10.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.460+0000 7f6c9effd640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:35:10.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.460+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb00abe70 con 0x7f6cb00a98f0 2024-03-25T19:35:10.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1148936741 0 0) 0x7f6c90004eb0 con 0x7f6cb01430c0 2024-03-25T19:35:10.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb0133ad0 con 0x7f6cb01430c0 2024-03-25T19:35:10.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6ca8001c90 con 0x7f6cb01430c0 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 10 monclient(hunting): dump: 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:10.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:10.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861374300 0 0) 0x7f6cb00ac150 con 0x7f6cb013f710 2024-03-25T19:35:10.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:10.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:10.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:10.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:35:10.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.466+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c90004eb0 con 0x7f6cb013f710 2024-03-25T19:35:10.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.475+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4020668266 0 0) 0x7f6cb00abe70 con 0x7f6cb00a98f0 2024-03-25T19:35:10.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.475+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb00ac150 con 0x7f6cb00a98f0 2024-03-25T19:35:10.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.475+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6ca0001930 con 0x7f6cb00a98f0 2024-03-25T19:35:10.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.475+0000 7f6c9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:10.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.475+0000 7f6c9effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:10.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.475+0000 7f6c9effd640 10 monclient(hunting): dump: 2024-03-25T19:35:10.475 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:10.475 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:10.475 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.475 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.476 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:10.476 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:10.476 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:10.476 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:10.476 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:10.476 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:10.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.479+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2850816713 0 0) 0x7f6c90004eb0 con 0x7f6cb013f710 2024-03-25T19:35:10.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.479+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb00abe70 con 0x7f6cb013f710 2024-03-25T19:35:10.478 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.479+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6ca4001b00 con 0x7f6cb013f710 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.479+0000 7f6c9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.479+0000 7f6c9effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.479+0000 7f6c9effd640 10 monclient(hunting): dump: 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:10.479 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:10.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1103786226 0 0) 0x7f6cb00ac150 con 0x7f6cb00a98f0 2024-03-25T19:35:10.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 >> v1:172.21.15.52:6790/0 conn(0x7f6cb013f710 legacy=0x7f6cb0132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:10.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 >> v1:172.21.15.179:6789/0 conn(0x7f6cb01430c0 legacy=0x7f6cb01333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:10.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6c9effd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:10.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6c9effd640 1 monclient: found mon.a 2024-03-25T19:35:10.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6c9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:10.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6c9effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:35:10.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb0139240 con 0x7f6cb00a98f0 2024-03-25T19:35:10.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6c9effd640 10 monclient: _finish_auth 0 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6c9effd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6c9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6cb4a93640 5 monclient: authenticate success, global_id 4569 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6cb4a93640 10 monclient: _renew_subs 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6cb4a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6cb4a93640 1 -- 172.21.15.52:0/2373681232 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb0137e50 con 0x7f6cb00a98f0 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6cb4a93640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6cb4a93640 10 monclient: _renew_subs 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6cb4a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6cb4a93640 1 -- 172.21.15.52:0/2373681232 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb0136680 con 0x7f6cb00a98f0 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.480+0000 7f6cb4a93640 1 librados: init done 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.481+0000 7f6cb4a93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.481+0000 7f6cb4a93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:10.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.481+0000 7f6cb4a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:35:10.483 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.481+0000 7f6cb4a93640 1 -- 172.21.15.52:0/2373681232 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb0074cc0 con 0x7f6cb00a98f0 2024-03-25T19:35:10.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.842+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ca00047e0 con 0x7f6cb00a98f0 2024-03-25T19:35:10.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.842+0000 7f6c9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:10.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.842+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6ca0004af0 con 0x7f6cb00a98f0 2024-03-25T19:35:10.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.842+0000 7f6c9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:10.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.842+0000 7f6c9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:10.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.842+0000 7f6c9effd640 10 monclient: dump: 2024-03-25T19:35:10.842 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:10.842 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:10.842 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.842 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:10.842 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:10.843 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:10.843 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:10.843 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:10.843 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:10.843 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:10.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.842+0000 7f6cacd99640 4 set_mon_vals no callback set 2024-03-25T19:35:10.843 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.843+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f6ca001aac0 con 0x7f6cb00a98f0 2024-03-25T19:35:10.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.843+0000 7f6cadd9b640 10 monclient: get_auth_request con 0x7f6c9006e130 auth_method 0 2024-03-25T19:35:10.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.843+0000 7f6c9effd640 10 client.4569.objecter ms_handle_connect 0x7f6c9006e130 2024-03-25T19:35:10.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.850+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f6ca0054730 con 0x7f6cb00a98f0 2024-03-25T19:35:10.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.850+0000 7f6c9effd640 10 client.4569.objecter ms_dispatch 0x7f6cb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:10.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.850+0000 7f6c9effd640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:10.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.850+0000 7f6c9effd640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:10.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.850+0000 7f6c9effd640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-25T19:35:10.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.850+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ca0050240 con 0x7f6cb00a98f0 2024-03-25T19:35:10.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.850+0000 7f6c9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:10.850 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:10.850+0000 7f6c9effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:11.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.076+0000 7f6cb4a93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.076+0000 7f6cb4a93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.076+0000 7f6cb4a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:35:11.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.076+0000 7f6cb4a93640 1 -- 172.21.15.52:0/2373681232 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cb00627b0 con 0x7f6cb00a98f0 2024-03-25T19:35:11.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.142+0000 7f6c9effd640 1 -- 172.21.15.52:0/2373681232 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f6ca0052170 con 0x7f6cb00a98f0 2024-03-25T19:35:11.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.142+0000 7f6c9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:11.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.142+0000 7f6c9effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:11.142 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:35:11.142 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:11.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 10 librados: watch_flush enter 2024-03-25T19:35:11.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 10 librados: watch_flush exit 2024-03-25T19:35:11.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-25T19:35:11.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 10 client.4569.objecter successfully canceled tick 2024-03-25T19:35:11.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 1 -- 172.21.15.52:0/2373681232 >> v1:172.21.15.52:6800/3967427147 conn(0x7f6c9006e130 legacy=0x7f6c900705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 10 monclient: shutdown 2024-03-25T19:35:11.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:11.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 1 -- 172.21.15.52:0/2373681232 >> v1:172.21.15.52:6789/0 conn(0x7f6cb00a98f0 legacy=0x7f6cb00ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:11.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 1 -- 172.21.15.52:0/2373681232 shutdown_connections 2024-03-25T19:35:11.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 1 -- 172.21.15.52:0/2373681232 >> 172.21.15.52:0/2373681232 conn(0x7f6cb00b7ea0 msgr2=0x7f6cb0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:11.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 1 -- 172.21.15.52:0/2373681232 shutdown_connections 2024-03-25T19:35:11.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.144+0000 7f6cb4a93640 1 -- 172.21.15.52:0/2373681232 wait complete. 2024-03-25T19:35:11.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:11.145+0000 7f6cb4a93640 1 librados: shutdown 2024-03-25T19:35:11.155 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:12.548 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:08:06.146 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.555 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:08:06.154 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.581 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:08:06.180 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.612 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:08:06.211 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.621 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:08:06.163 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.623 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:08:06.222 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.636 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:08:06.177 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.659 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:08:06.258 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:12.663 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:08:06.205 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:17.962 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.076+0000 7f85fd354640 10 monclient: get_monmap_and_config 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.076+0000 7f85fd354640 10 monclient: build_initial_monmap 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.076+0000 7f85fd354640 10 monclient: monmap: 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:35:18.077091+0000 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:35:18.077091+0000 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:35:18.076 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:35:18.077 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.077+0000 7f85fd354640 1 Processor -- start 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.077+0000 7f85fd354640 1 -- start start 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.077+0000 7f85fd354640 10 monclient: init 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.078+0000 7f85fd354640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.078+0000 7f85fd354640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.078+0000 7f85fd354640 10 monclient(hunting): picked mon.noname-c con 0x7f85f80a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.078+0000 7f85fd354640 10 monclient(hunting): picked mon.noname-a con 0x7f85f813f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.078+0000 7f85fd354640 10 monclient(hunting): picked mon.noname-b con 0x7f85f81430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.078+0000 7f85fd354640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f8142520 con 0x7f85f813f710 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.078+0000 7f85fd354640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f80b28a0 con 0x7f85f80a98f0 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.078+0000 7f85fd354640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f80b2a30 con 0x7f85f81430c0 2024-03-25T19:35:18.078 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.078+0000 7f85fd354640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:18.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.078+0000 7f85fd354640 10 monclient(hunting): authenticate will time out at 6366.258073s 2024-03-25T19:35:18.079 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.080+0000 7f85f6ffd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f85f80a98f0 0x7f85f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:58340/0 (socket says 172.21.15.52:58340) 2024-03-25T19:35:18.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.080+0000 7f85f6ffd640 1 -- 172.21.15.52:0/1650469965 learned_addr learned my addr 172.21.15.52:0/1650469965 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:18.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.087+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023190303 0 0) 0x7f85f80b28a0 con 0x7f85f80a98f0 2024-03-25T19:35:18.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.087+0000 7f85f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.087+0000 7f85f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.087+0000 7f85f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.087+0000 7f85f5ffb640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:35:18.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.087+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e0003520 con 0x7f85f80a98f0 2024-03-25T19:35:18.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.097+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447133596 0 0) 0x7f85f8142520 con 0x7f85f813f710 2024-03-25T19:35:18.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.097+0000 7f85f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.097+0000 7f85f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.097+0000 7f85f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.097+0000 7f85f5ffb640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:35:18.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.097+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f80b28a0 con 0x7f85f813f710 2024-03-25T19:35:18.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.102+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168182384 0 0) 0x7f85f80b2a30 con 0x7f85f81430c0 2024-03-25T19:35:18.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.102+0000 7f85f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.102+0000 7f85f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.102+0000 7f85f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.102+0000 7f85f5ffb640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:35:18.102 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.102+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f8142520 con 0x7f85f81430c0 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.102+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2063165622 0 0) 0x7f85e0003520 con 0x7f85f80a98f0 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.102+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 >> v1:172.21.15.52:6790/0 conn(0x7f85f81430c0 legacy=0x7f85f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.102+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 >> v1:172.21.15.52:6789/0 conn(0x7f85f813f710 legacy=0x7f85f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.102+0000 7f85f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.102+0000 7f85f5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85f5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f80b44c0 con 0x7f85f80a98f0 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85f5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:18.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f85e8001690 con 0x7f85f80a98f0 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85fd354640 5 monclient: authenticate success, global_id 4585 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85fd354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85f5ffb640 10 monclient: dump: 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:18.104 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:18.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.103+0000 7f85fd354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:18.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.126+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85e80017f0 con 0x7f85f80a98f0 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.126+0000 7f85f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.126+0000 7f85f5ffb640 1 -- 172.21.15.52:0/1650469965 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f85e8001c00 con 0x7f85f80a98f0 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.126+0000 7f85fd354640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.126+0000 7f85fd354640 4 set_mon_vals no callback set 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.126+0000 7f85f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.126+0000 7f85f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.126+0000 7f85f5ffb640 10 monclient: dump: 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.126 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:18.127 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:18.127 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:18.127 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:18.127 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:18.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.127+0000 7f85fd354640 10 monclient: shutdown 2024-03-25T19:35:18.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.127+0000 7f85fd354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:18.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.127+0000 7f85fd354640 1 -- 172.21.15.52:0/1650469965 >> v1:172.21.15.179:6789/0 conn(0x7f85f80a98f0 legacy=0x7f85f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.127+0000 7f85fd354640 1 -- 172.21.15.52:0/1650469965 shutdown_connections 2024-03-25T19:35:18.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.127+0000 7f85fd354640 1 -- 172.21.15.52:0/1650469965 >> 172.21.15.52:0/1650469965 conn(0x7f85f80b7ea0 msgr2=0x7f85f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:18.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.127+0000 7f85fd354640 1 -- 172.21.15.52:0/1650469965 shutdown_connections 2024-03-25T19:35:18.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.127+0000 7f85fd354640 1 -- 172.21.15.52:0/1650469965 wait complete. 2024-03-25T19:35:18.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.128+0000 7f85fd354640 10 monclient: build_initial_monmap 2024-03-25T19:35:18.129 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.128+0000 7f85fd354640 10 monclient: monmap: 2024-03-25T19:35:18.129 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:18.129 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.128+0000 7f85fd354640 1 librados: starting msgr at 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.128+0000 7f85fd354640 1 librados: starting objecter 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.128+0000 7f85fd354640 1 Processor -- start 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.128+0000 7f85fd354640 1 -- start start 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.128+0000 7f85fd354640 1 librados: setting wanted keys 2024-03-25T19:35:18.130 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.128+0000 7f85fd354640 1 librados: calling monclient init 2024-03-25T19:35:18.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.128+0000 7f85fd354640 10 monclient: init 2024-03-25T19:35:18.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.129+0000 7f85fd354640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:18.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.129+0000 7f85fd354640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:18.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.129+0000 7f85fd354640 10 monclient(hunting): picked mon.noname-b con 0x7f85f80a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:18.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.129+0000 7f85fd354640 10 monclient(hunting): picked mon.noname-c con 0x7f85f813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:18.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.129+0000 7f85fd354640 10 monclient(hunting): picked mon.noname-a con 0x7f85f81430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:18.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.129+0000 7f85fd354640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f8138a50 con 0x7f85f81430c0 2024-03-25T19:35:18.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.129+0000 7f85fd354640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f80ab9e0 con 0x7f85f80a98f0 2024-03-25T19:35:18.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.129+0000 7f85fd354640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f81d7880 con 0x7f85f813f710 2024-03-25T19:35:18.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.129+0000 7f85fd354640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:18.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.129+0000 7f85fd354640 10 monclient(hunting): authenticate will time out at 6366.309502s 2024-03-25T19:35:18.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.134+0000 7f85f6ffd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f85f80a98f0 0x7f85f8138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:58372/0 (socket says 172.21.15.52:58372) 2024-03-25T19:35:18.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.134+0000 7f85f6ffd640 1 -- 172.21.15.52:0/2503621413 learned_addr learned my addr 172.21.15.52:0/2503621413 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:18.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.135+0000 7f85df7fe640 10 client.?.objecter ms_handle_connect 0x7f85f80a98f0 2024-03-25T19:35:18.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.135+0000 7f85df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.142+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548204413 0 0) 0x7f85f80ab9e0 con 0x7f85f80a98f0 2024-03-25T19:35:18.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.142+0000 7f85df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.142 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.142+0000 7f85df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.142+0000 7f85df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.142+0000 7f85df7fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:35:18.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.142+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e4004eb0 con 0x7f85f80a98f0 2024-03-25T19:35:18.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.142+0000 7f85df7fe640 10 client.?.objecter ms_handle_connect 0x7f85f813f710 2024-03-25T19:35:18.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.142+0000 7f85df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.143+0000 7f85df7fe640 10 client.?.objecter ms_handle_connect 0x7f85f81430c0 2024-03-25T19:35:18.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.143+0000 7f85df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:18.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.155+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760235958 0 0) 0x7f85f8138a50 con 0x7f85f81430c0 2024-03-25T19:35:18.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.155+0000 7f85df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.155+0000 7f85df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.155+0000 7f85df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.155+0000 7f85df7fe640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:35:18.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.155+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f80ab9e0 con 0x7f85f81430c0 2024-03-25T19:35:18.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.160+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353981617 0 0) 0x7f85f81d7880 con 0x7f85f813f710 2024-03-25T19:35:18.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.160+0000 7f85df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:18.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.160+0000 7f85df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:18.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.160+0000 7f85df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:18.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.160+0000 7f85df7fe640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:35:18.160 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.160+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f8138a50 con 0x7f85f813f710 2024-03-25T19:35:18.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.161+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3581364725 0 0) 0x7f85e4004eb0 con 0x7f85f80a98f0 2024-03-25T19:35:18.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.161+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85f81d7880 con 0x7f85f80a98f0 2024-03-25T19:35:18.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.161+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f85e8001bc0 con 0x7f85f80a98f0 2024-03-25T19:35:18.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.161+0000 7f85df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.161+0000 7f85df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:18.161 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.161+0000 7f85df7fe640 10 monclient(hunting): dump: 2024-03-25T19:35:18.161 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:18.161 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:18.161 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:18.161 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:18.161 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.162 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:18.162 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:18.162 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:18.162 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:18.162 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:18.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.161+0000 7f85df7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:18.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4236734638 0 0) 0x7f85f81d7880 con 0x7f85f80a98f0 2024-03-25T19:35:18.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 >> v1:172.21.15.52:6790/0 conn(0x7f85f813f710 legacy=0x7f85f81d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 >> v1:172.21.15.52:6789/0 conn(0x7f85f81430c0 legacy=0x7f85f81e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85df7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85df7fe640 1 monclient: found mon.b 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85df7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f81d84b0 con 0x7f85f80a98f0 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85df7fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85df7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85fd354640 5 monclient: authenticate success, global_id 4588 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85fd354640 10 monclient: _renew_subs 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85fd354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85fd354640 1 -- 172.21.15.52:0/2503621413 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f81d51f0 con 0x7f85f80a98f0 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85fd354640 10 client.4588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85fd354640 10 monclient: _renew_subs 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85fd354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:35:18.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85fd354640 1 -- 172.21.15.52:0/2503621413 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f8138e60 con 0x7f85f80a98f0 2024-03-25T19:35:18.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.172+0000 7f85fd354640 1 librados: init done 2024-03-25T19:35:18.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.174+0000 7f85fd354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:18.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.174+0000 7f85fd354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:18.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.174+0000 7f85fd354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:35:18.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.174+0000 7f85fd354640 1 -- 172.21.15.52:0/2503621413 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f81d76e0 con 0x7f85f80a98f0 2024-03-25T19:35:18.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.466+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85e80011f0 con 0x7f85f80a98f0 2024-03-25T19:35:18.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.466+0000 7f85df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:18.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.466+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f85e8004b10 con 0x7f85f80a98f0 2024-03-25T19:35:18.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.466+0000 7f85df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:18.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.466+0000 7f85df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:18.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.466+0000 7f85df7fe640 10 monclient: dump: 2024-03-25T19:35:18.466 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:18.466 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:18.466 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:18.466 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:18.467 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:18.467 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:18.467 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:18.467 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:18.467 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:18.467 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:18.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.466+0000 7f85f57fa640 4 set_mon_vals no callback set 2024-03-25T19:35:18.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.466+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f85e801ab60 con 0x7f85f80a98f0 2024-03-25T19:35:18.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.467+0000 7f85f67fc640 10 monclient: get_auth_request con 0x7f85e406dd50 auth_method 0 2024-03-25T19:35:18.468 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.467+0000 7f85df7fe640 10 client.4588.objecter ms_handle_connect 0x7f85e406dd50 2024-03-25T19:35:18.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.471+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f85e80547d0 con 0x7f85f80a98f0 2024-03-25T19:35:18.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.471+0000 7f85df7fe640 10 client.4588.objecter ms_dispatch 0x7f85f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:18.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.471+0000 7f85df7fe640 3 client.4588.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:18.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.471+0000 7f85df7fe640 3 client.4588.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:18.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.471+0000 7f85df7fe640 20 client.4588.objecter dump_active .. 0 homeless 2024-03-25T19:35:18.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.471+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85e8050110 con 0x7f85f80a98f0 2024-03-25T19:35:18.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.471+0000 7f85df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:18.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.471+0000 7f85df7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:18.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.726+0000 7f85fd354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:18.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.726+0000 7f85fd354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:18.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.726+0000 7f85fd354640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:35:18.726 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.726+0000 7f85fd354640 1 -- 172.21.15.52:0/2503621413 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85f80627b0 con 0x7f85f80a98f0 2024-03-25T19:35:18.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.751+0000 7f85df7fe640 1 -- 172.21.15.52:0/2503621413 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f85e8052210 con 0x7f85f80a98f0 2024-03-25T19:35:18.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.751+0000 7f85df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:18.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.751+0000 7f85df7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:18.751 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:35:18.752 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:18.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 10 librados: watch_flush enter 2024-03-25T19:35:18.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 10 librados: watch_flush exit 2024-03-25T19:35:18.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 20 client.4588.objecter shutdown clearing up homeless session... 2024-03-25T19:35:18.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 10 client.4588.objecter successfully canceled tick 2024-03-25T19:35:18.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 1 -- 172.21.15.52:0/2503621413 >> v1:172.21.15.52:6800/3967427147 conn(0x7f85e406dd50 legacy=0x7f85e4070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 10 monclient: shutdown 2024-03-25T19:35:18.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:18.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 1 -- 172.21.15.52:0/2503621413 >> v1:172.21.15.179:6789/0 conn(0x7f85f80a98f0 legacy=0x7f85f8138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:18.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 1 -- 172.21.15.52:0/2503621413 shutdown_connections 2024-03-25T19:35:18.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 1 -- 172.21.15.52:0/2503621413 >> 172.21.15.52:0/2503621413 conn(0x7f85f80b7ea0 msgr2=0x7f85f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:18.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 1 -- 172.21.15.52:0/2503621413 shutdown_connections 2024-03-25T19:35:18.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 1 -- 172.21.15.52:0/2503621413 wait complete. 2024-03-25T19:35:18.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:18.754+0000 7f85fd354640 1 librados: shutdown 2024-03-25T19:35:18.767 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:25.574 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:25.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.687+0000 7ff5f99f4640 10 monclient: get_monmap_and_config 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.687+0000 7ff5f99f4640 10 monclient: build_initial_monmap 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.687+0000 7ff5f99f4640 10 monclient: monmap: 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:35:25.688067+0000 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:35:25.688067+0000 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:35:25.687 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:25.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.688+0000 7ff5f99f4640 1 Processor -- start 2024-03-25T19:35:25.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.688+0000 7ff5f99f4640 1 -- start start 2024-03-25T19:35:25.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.688+0000 7ff5f99f4640 10 monclient: init 2024-03-25T19:35:25.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.689+0000 7ff5f99f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:25.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.689+0000 7ff5f99f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:25.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.689+0000 7ff5f99f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff5f406fb30 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:25.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.689+0000 7ff5f99f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff5f413c0b0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:25.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.689+0000 7ff5f99f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff5f406ff10 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:25.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.689+0000 7ff5f99f4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f4071450 con 0x7ff5f413c0b0 2024-03-25T19:35:25.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.689+0000 7ff5f99f4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f4071730 con 0x7ff5f406fb30 2024-03-25T19:35:25.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.689+0000 7ff5f99f4640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f40730f0 con 0x7ff5f406ff10 2024-03-25T19:35:25.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.689+0000 7ff5f99f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:25.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.689+0000 7ff5f99f4640 10 monclient(hunting): authenticate will time out at 6373.869498s 2024-03-25T19:35:25.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.691+0000 7ff5f37fe640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7ff5f406ff10 0x7ff5f40615c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:42756/0 (socket says 172.21.15.52:42756) 2024-03-25T19:35:25.690 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.691+0000 7ff5f37fe640 1 -- 172.21.15.52:0/1544609346 learned_addr learned my addr 172.21.15.52:0/1544609346 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.705+0000 7ff5f1ffb640 1 -- 172.21.15.52:0/1544609346 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264806505 0 0) 0x7ff5f40730f0 con 0x7ff5f406ff10 2024-03-25T19:35:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.705+0000 7ff5f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.705+0000 7ff5f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.705+0000 7ff5f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.705+0000 7ff5f1ffb640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:35:25.705 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.705+0000 7ff5f1ffb640 1 -- 172.21.15.52:0/1544609346 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5dc003520 con 0x7ff5f406ff10 2024-03-25T19:35:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.714+0000 7ff5f1ffb640 1 -- 172.21.15.52:0/1544609346 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038330039 0 0) 0x7ff5f4071450 con 0x7ff5f413c0b0 2024-03-25T19:35:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.714+0000 7ff5f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.714+0000 7ff5f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.714+0000 7ff5f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.714+0000 7ff5f1ffb640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:35:25.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.714+0000 7ff5f1ffb640 1 -- 172.21.15.52:0/1544609346 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f40730f0 con 0x7ff5f413c0b0 2024-03-25T19:35:25.718 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.718+0000 7ff5f1ffb640 1 -- 172.21.15.52:0/1544609346 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 553577650 0 0) 0x7ff5dc003520 con 0x7ff5f406ff10 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 1 -- 172.21.15.52:0/1544609346 >> v1:172.21.15.179:6789/0 conn(0x7ff5f406fb30 legacy=0x7ff5f413b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 1 -- 172.21.15.52:0/1544609346 >> v1:172.21.15.52:6789/0 conn(0x7ff5f413c0b0 legacy=0x7ff5f4071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 1 monclient: found mon.noname-b 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 1 -- 172.21.15.52:0/1544609346 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f406ef90 con 0x7ff5f406ff10 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 1 -- 172.21.15.52:0/1544609346 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff5ec001a10 con 0x7ff5f406ff10 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f1ffb640 10 monclient: dump: 2024-03-25T19:35:25.719 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:25.720 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:25.720 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:25.720 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:25.720 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:25.720 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:25.720 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:25.720 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:25.720 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:25.720 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:25.720 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f99f4640 5 monclient: authenticate success, global_id 4577 2024-03-25T19:35:25.720 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.719+0000 7ff5f99f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:25.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f1ffb640 1 -- 172.21.15.52:0/1544609346 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5ec001b70 con 0x7ff5f406ff10 2024-03-25T19:35:25.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:25.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f1ffb640 1 -- 172.21.15.52:0/1544609346 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff5ec001f80 con 0x7ff5f406ff10 2024-03-25T19:35:25.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f99f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f99f4640 4 set_mon_vals no callback set 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f1ffb640 10 monclient: dump: 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:25.728 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:25.729 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f99f4640 10 monclient: shutdown 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f99f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f99f4640 1 -- 172.21.15.52:0/1544609346 >> v1:172.21.15.52:6790/0 conn(0x7ff5f406ff10 legacy=0x7ff5f40615c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f99f4640 1 -- 172.21.15.52:0/1544609346 shutdown_connections 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f99f4640 1 -- 172.21.15.52:0/1544609346 >> 172.21.15.52:0/1544609346 conn(0x7ff5f40b11e0 msgr2=0x7ff5f40b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.728+0000 7ff5f99f4640 1 -- 172.21.15.52:0/1544609346 shutdown_connections 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.729+0000 7ff5f99f4640 1 -- 172.21.15.52:0/1544609346 wait complete. 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.729+0000 7ff5f99f4640 10 monclient: build_initial_monmap 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.729+0000 7ff5f99f4640 10 monclient: monmap: 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:25.730 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:35:25.731 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:25.731 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:35:25.731 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:35:25.731 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:25.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.730+0000 7ff5f99f4640 1 librados: starting msgr at 2024-03-25T19:35:25.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.730+0000 7ff5f99f4640 1 librados: starting objecter 2024-03-25T19:35:25.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.730+0000 7ff5f99f4640 1 Processor -- start 2024-03-25T19:35:25.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.730+0000 7ff5f99f4640 1 -- start start 2024-03-25T19:35:25.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.730+0000 7ff5f99f4640 1 librados: setting wanted keys 2024-03-25T19:35:25.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.730+0000 7ff5f99f4640 1 librados: calling monclient init 2024-03-25T19:35:25.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.730+0000 7ff5f99f4640 10 monclient: init 2024-03-25T19:35:25.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.730+0000 7ff5f99f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:25.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.730+0000 7ff5f99f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:25.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.731+0000 7ff5f99f4640 10 monclient(hunting): picked mon.noname-c con 0x7ff5f406fb30 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:25.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.731+0000 7ff5f99f4640 10 monclient(hunting): picked mon.noname-b con 0x7ff5f406ff10 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:25.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.731+0000 7ff5f99f4640 10 monclient(hunting): picked mon.noname-a con 0x7ff5f413c0b0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:25.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.731+0000 7ff5f99f4640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f41df020 con 0x7ff5f413c0b0 2024-03-25T19:35:25.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.731+0000 7ff5f99f4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f41debf0 con 0x7ff5f406ff10 2024-03-25T19:35:25.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.731+0000 7ff5f99f4640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f40aaba0 con 0x7ff5f406fb30 2024-03-25T19:35:25.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.731+0000 7ff5f99f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:25.732 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.731+0000 7ff5f99f4640 10 monclient(hunting): authenticate will time out at 6373.911019s 2024-03-25T19:35:25.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.742+0000 7ff5f2ffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7ff5f406fb30 0x7ff5f40ac170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:42802/0 (socket says 172.21.15.52:42802) 2024-03-25T19:35:25.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.742+0000 7ff5f2ffd640 1 -- 172.21.15.52:0/2327663879 learned_addr learned my addr 172.21.15.52:0/2327663879 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:25.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.743+0000 7ff5dbfff640 10 client.?.objecter ms_handle_connect 0x7ff5f406fb30 2024-03-25T19:35:25.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.743+0000 7ff5dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:25.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.747+0000 7ff5dbfff640 10 client.?.objecter ms_handle_connect 0x7ff5f413c0b0 2024-03-25T19:35:25.747 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.748+0000 7ff5dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:25.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.756+0000 7ff5dbfff640 10 client.?.objecter ms_handle_connect 0x7ff5f406ff10 2024-03-25T19:35:25.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.756+0000 7ff5dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:25.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.764+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92413014 0 0) 0x7ff5f40aaba0 con 0x7ff5f406fb30 2024-03-25T19:35:25.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.764+0000 7ff5dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:25.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.764+0000 7ff5dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:25.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.764+0000 7ff5dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:25.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.764+0000 7ff5dbfff640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:35:25.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.764+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e0004eb0 con 0x7ff5f406fb30 2024-03-25T19:35:25.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.765+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742495557 0 0) 0x7ff5f41debf0 con 0x7ff5f406ff10 2024-03-25T19:35:25.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.765+0000 7ff5dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:25.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.765+0000 7ff5dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:25.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.765+0000 7ff5dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:25.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.765+0000 7ff5dbfff640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:35:25.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.765+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f40aaba0 con 0x7ff5f406ff10 2024-03-25T19:35:25.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.769+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634064023 0 0) 0x7ff5f41df020 con 0x7ff5f413c0b0 2024-03-25T19:35:25.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.769+0000 7ff5dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:25.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.769+0000 7ff5dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:25.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.769+0000 7ff5dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:25.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.769+0000 7ff5dbfff640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:35:25.769 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.769+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f41debf0 con 0x7ff5f413c0b0 2024-03-25T19:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.783+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3190889762 0 0) 0x7ff5e0004eb0 con 0x7ff5f406fb30 2024-03-25T19:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.783+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5f41df020 con 0x7ff5f406fb30 2024-03-25T19:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.783+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff5e40019d0 con 0x7ff5f406fb30 2024-03-25T19:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.783+0000 7ff5dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.783+0000 7ff5dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.783+0000 7ff5dbfff640 10 monclient(hunting): dump: 2024-03-25T19:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:25.783 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:25.784 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:25.784 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.783+0000 7ff5dbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:25.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1950676552 0 0) 0x7ff5f41df020 con 0x7ff5f406fb30 2024-03-25T19:35:25.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 >> v1:172.21.15.179:6789/0 conn(0x7ff5f406ff10 legacy=0x7ff5f41df4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:25.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 >> v1:172.21.15.52:6789/0 conn(0x7ff5f413c0b0 legacy=0x7ff5f40aa490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:25.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5dbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:25.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5dbfff640 1 monclient: found mon.c 2024-03-25T19:35:25.792 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5dbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f40aad30 con 0x7ff5f406fb30 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5dbfff640 10 monclient: _finish_auth 0 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5dbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5f99f4640 5 monclient: authenticate success, global_id 4580 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5f99f4640 10 monclient: _renew_subs 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5f99f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5f99f4640 1 -- 172.21.15.52:0/2327663879 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f40a9db0 con 0x7ff5f406fb30 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5f99f4640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5f99f4640 10 monclient: _renew_subs 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5f99f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:25.793 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5f99f4640 1 -- 172.21.15.52:0/2327663879 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f40acde0 con 0x7ff5f406fb30 2024-03-25T19:35:25.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.792+0000 7ff5f99f4640 1 librados: init done 2024-03-25T19:35:25.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.793+0000 7ff5f99f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:25.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.793+0000 7ff5f99f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:25.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.793+0000 7ff5f99f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:25.794 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:25.793+0000 7ff5f99f4640 1 -- 172.21.15.52:0/2327663879 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f41ded80 con 0x7ff5f406fb30 2024-03-25T19:35:26.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.119+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5e40035f0 con 0x7ff5f406fb30 2024-03-25T19:35:26.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.119+0000 7ff5dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:26.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.119+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff5e4003980 con 0x7ff5f406fb30 2024-03-25T19:35:26.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.119+0000 7ff5dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:26.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.119+0000 7ff5dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:26.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.119+0000 7ff5dbfff640 10 monclient: dump: 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.119+0000 7ff5f17fa640 4 set_mon_vals no callback set 2024-03-25T19:35:26.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.120+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7ff5e4003ba0 con 0x7ff5f406fb30 2024-03-25T19:35:26.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.120+0000 7ff5f27fc640 10 monclient: get_auth_request con 0x7ff5e006ddf0 auth_method 0 2024-03-25T19:35:26.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.120+0000 7ff5dbfff640 10 client.4580.objecter ms_handle_connect 0x7ff5e006ddf0 2024-03-25T19:35:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.126+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7ff5e4054810 con 0x7ff5f406fb30 2024-03-25T19:35:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.126+0000 7ff5dbfff640 10 client.4580.objecter ms_dispatch 0x7ff5f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.126+0000 7ff5dbfff640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.126+0000 7ff5dbfff640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.126+0000 7ff5dbfff640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-25T19:35:26.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.126+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5e4050320 con 0x7ff5f406fb30 2024-03-25T19:35:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.126+0000 7ff5dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:26.127 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.126+0000 7ff5dbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:26.382 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.383+0000 7ff5f99f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:26.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.383+0000 7ff5f99f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:26.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.383+0000 7ff5f99f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:26.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.383+0000 7ff5f99f4640 1 -- 172.21.15.52:0/2327663879 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5f40627b0 con 0x7ff5f406fb30 2024-03-25T19:35:26.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.527+0000 7ff5dbfff640 1 -- 172.21.15.52:0/2327663879 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7ff5e4052250 con 0x7ff5f406fb30 2024-03-25T19:35:26.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.527+0000 7ff5dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:26.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.527+0000 7ff5dbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:26.527 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:35:26.527 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:26.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.529+0000 7ff5f99f4640 10 librados: watch_flush enter 2024-03-25T19:35:26.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.529+0000 7ff5f99f4640 10 librados: watch_flush exit 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.529+0000 7ff5f99f4640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.529+0000 7ff5f99f4640 10 client.4580.objecter successfully canceled tick 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.529+0000 7ff5f99f4640 1 -- 172.21.15.52:0/2327663879 >> v1:172.21.15.52:6800/3967427147 conn(0x7ff5e006ddf0 legacy=0x7ff5e00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.529+0000 7ff5f99f4640 10 monclient: shutdown 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.529+0000 7ff5f99f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.529+0000 7ff5f99f4640 1 -- 172.21.15.52:0/2327663879 >> v1:172.21.15.52:6790/0 conn(0x7ff5f406fb30 legacy=0x7ff5f40ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.530+0000 7ff5f99f4640 1 -- 172.21.15.52:0/2327663879 shutdown_connections 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.530+0000 7ff5f99f4640 1 -- 172.21.15.52:0/2327663879 >> 172.21.15.52:0/2327663879 conn(0x7ff5f40b11e0 msgr2=0x7ff5f40ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.530+0000 7ff5f99f4640 1 -- 172.21.15.52:0/2327663879 shutdown_connections 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.530+0000 7ff5f99f4640 1 -- 172.21.15.52:0/2327663879 wait complete. 2024-03-25T19:35:26.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:26.530+0000 7ff5f99f4640 1 librados: shutdown 2024-03-25T19:35:26.541 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:33.348 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:33.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.464+0000 7fed08f9a640 10 monclient: get_monmap_and_config 2024-03-25T19:35:33.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.464+0000 7fed08f9a640 10 monclient: build_initial_monmap 2024-03-25T19:35:33.464 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.464+0000 7fed08f9a640 10 monclient: monmap: 2024-03-25T19:35:33.464 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:33.464 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:33.464 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:35:33.465123+0000 2024-03-25T19:35:33.464 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:35:33.465123+0000 2024-03-25T19:35:33.464 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:33.464 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:33.465 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:33.465 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:35:33.465 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:35:33.465 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:33.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.465+0000 7fed08f9a640 1 Processor -- start 2024-03-25T19:35:33.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.465+0000 7fed08f9a640 1 -- start start 2024-03-25T19:35:33.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.465+0000 7fed08f9a640 10 monclient: init 2024-03-25T19:35:33.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.466+0000 7fed08f9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:33.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.466+0000 7fed08f9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:33.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.466+0000 7fed08f9a640 10 monclient(hunting): picked mon.noname-c con 0x7fed040b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:33.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.466+0000 7fed08f9a640 10 monclient(hunting): picked mon.noname-b con 0x7fed040b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:33.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.466+0000 7fed08f9a640 10 monclient(hunting): picked mon.noname-a con 0x7fed0414be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:33.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.466+0000 7fed08f9a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed040af940 con 0x7fed0414be90 2024-03-25T19:35:33.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.466+0000 7fed08f9a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed040b0a70 con 0x7fed040b4d60 2024-03-25T19:35:33.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.466+0000 7fed08f9a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed040b0d50 con 0x7fed040b1730 2024-03-25T19:35:33.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.466+0000 7fed08f9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:33.467 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.466+0000 7fed08f9a640 10 monclient(hunting): authenticate will time out at 6381.646646s 2024-03-25T19:35:33.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.469+0000 7fed0259c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fed040b4d60 0x7fed0413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:60774/0 (socket says 172.21.15.52:60774) 2024-03-25T19:35:33.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.469+0000 7fed0259c640 1 -- 172.21.15.52:0/2421951320 learned_addr learned my addr 172.21.15.52:0/2421951320 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:33.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.493+0000 7fed01d9b640 1 -- 172.21.15.52:0/2421951320 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772377033 0 0) 0x7fed040b0a70 con 0x7fed040b4d60 2024-03-25T19:35:33.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.493+0000 7fed01d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.493+0000 7fed01d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.493+0000 7fed01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.493+0000 7fed01d9b640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:35:33.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.493+0000 7fed01d9b640 1 -- 172.21.15.52:0/2421951320 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf0003520 con 0x7fed040b4d60 2024-03-25T19:35:33.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.494+0000 7fed01d9b640 1 -- 172.21.15.52:0/2421951320 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865508778 0 0) 0x7fed040af940 con 0x7fed0414be90 2024-03-25T19:35:33.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.494+0000 7fed01d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.494+0000 7fed01d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.494+0000 7fed01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.494+0000 7fed01d9b640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:35:33.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.494+0000 7fed01d9b640 1 -- 172.21.15.52:0/2421951320 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed040b0a70 con 0x7fed0414be90 2024-03-25T19:35:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 1 -- 172.21.15.52:0/2421951320 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1098615968 0 0) 0x7fecf0003520 con 0x7fed040b4d60 2024-03-25T19:35:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 1 -- 172.21.15.52:0/2421951320 >> v1:172.21.15.52:6790/0 conn(0x7fed040b1730 legacy=0x7fed040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 1 -- 172.21.15.52:0/2421951320 >> v1:172.21.15.52:6789/0 conn(0x7fed0414be90 legacy=0x7fed0414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:35:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 1 monclient: found mon.noname-c 2024-03-25T19:35:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:33.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 1 -- 172.21.15.52:0/2421951320 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed040b1030 con 0x7fed040b4d60 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 10 monclient: _finish_auth 0 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 1 -- 172.21.15.52:0/2421951320 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fecf4001690 con 0x7fed040b4d60 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed08f9a640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed08f9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed01d9b640 10 monclient: dump: 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.500 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.501 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.501 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:33.501 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:33.501 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:33.501 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:33.501 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:33.501 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.499+0000 7fed08f9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.514+0000 7fed01d9b640 1 -- 172.21.15.52:0/2421951320 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecf40017f0 con 0x7fed040b4d60 2024-03-25T19:35:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.514+0000 7fed01d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.514+0000 7fed01d9b640 1 -- 172.21.15.52:0/2421951320 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fecf4001c00 con 0x7fed040b4d60 2024-03-25T19:35:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.514+0000 7fed01d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.514+0000 7fed01d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:33.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.514+0000 7fed01d9b640 10 monclient: dump: 2024-03-25T19:35:33.514 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:33.514 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:33.514 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.514 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.515 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.515 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:33.515 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:33.515 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:33.515 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:33.515 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:33.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.514+0000 7fed08f9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:33.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.514+0000 7fed08f9a640 4 set_mon_vals no callback set 2024-03-25T19:35:33.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.515+0000 7fed08f9a640 10 monclient: shutdown 2024-03-25T19:35:33.516 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.515+0000 7fed08f9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.515+0000 7fed08f9a640 1 -- 172.21.15.52:0/2421951320 >> v1:172.21.15.179:6789/0 conn(0x7fed040b4d60 legacy=0x7fed0413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.515+0000 7fed08f9a640 1 -- 172.21.15.52:0/2421951320 shutdown_connections 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.515+0000 7fed08f9a640 1 -- 172.21.15.52:0/2421951320 >> 172.21.15.52:0/2421951320 conn(0x7fed040617a0 msgr2=0x7fed04061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.515+0000 7fed08f9a640 1 -- 172.21.15.52:0/2421951320 shutdown_connections 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.515+0000 7fed08f9a640 1 -- 172.21.15.52:0/2421951320 wait complete. 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.516+0000 7fed08f9a640 10 monclient: build_initial_monmap 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.516+0000 7fed08f9a640 10 monclient: monmap: 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:35:33.517 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.517+0000 7fed08f9a640 1 librados: starting msgr at 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.517+0000 7fed08f9a640 1 librados: starting objecter 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.517+0000 7fed08f9a640 1 Processor -- start 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.517+0000 7fed08f9a640 1 -- start start 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.517+0000 7fed08f9a640 1 librados: setting wanted keys 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.517+0000 7fed08f9a640 1 librados: calling monclient init 2024-03-25T19:35:33.518 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.517+0000 7fed08f9a640 10 monclient: init 2024-03-25T19:35:33.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.518+0000 7fed08f9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:33.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.518+0000 7fed08f9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.518+0000 7fed08f9a640 10 monclient(hunting): picked mon.noname-c con 0x7fed040b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.518+0000 7fed08f9a640 10 monclient(hunting): picked mon.noname-b con 0x7fed040b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.518+0000 7fed08f9a640 10 monclient(hunting): picked mon.noname-a con 0x7fed0414be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.518+0000 7fed08f9a640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0413a8d0 con 0x7fed0414be90 2024-03-25T19:35:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.518+0000 7fed08f9a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed041d7840 con 0x7fed040b4d60 2024-03-25T19:35:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.518+0000 7fed08f9a640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed041d8470 con 0x7fed040b1730 2024-03-25T19:35:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.518+0000 7fed08f9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:33.520 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.518+0000 7fed08f9a640 10 monclient(hunting): authenticate will time out at 6381.698268s 2024-03-25T19:35:33.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.525+0000 7fed02d9d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fed0414be90 0x7fed041e21e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:58834/0 (socket says 172.21.15.52:58834) 2024-03-25T19:35:33.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.525+0000 7fed02d9d640 1 -- 172.21.15.52:0/2100400889 learned_addr learned my addr 172.21.15.52:0/2100400889 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:33.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.528+0000 7fecfaffd640 10 client.?.objecter ms_handle_connect 0x7fed0414be90 2024-03-25T19:35:33.528 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.528+0000 7fecfaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:33.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.542+0000 7fecfaffd640 10 client.?.objecter ms_handle_connect 0x7fed040b4d60 2024-03-25T19:35:33.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.542+0000 7fecfaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:33.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.542+0000 7fecfaffd640 10 client.?.objecter ms_handle_connect 0x7fed040b1730 2024-03-25T19:35:33.542 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.542+0000 7fecfaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:33.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.548+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069962590 0 0) 0x7fed0413a8d0 con 0x7fed0414be90 2024-03-25T19:35:33.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.548+0000 7fecfaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.548+0000 7fecfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.548+0000 7fecfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.548+0000 7fecfaffd640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:35:33.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.548+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece4004eb0 con 0x7fed0414be90 2024-03-25T19:35:33.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.556+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079424861 0 0) 0x7fed041d8470 con 0x7fed040b1730 2024-03-25T19:35:33.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.556+0000 7fecfaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.556+0000 7fecfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.556+0000 7fecfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.556+0000 7fecfaffd640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:35:33.556 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.556+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed0413a8d0 con 0x7fed040b1730 2024-03-25T19:35:33.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.559+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245885998 0 0) 0x7fed041d7840 con 0x7fed040b4d60 2024-03-25T19:35:33.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.559+0000 7fecfaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:33.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.559+0000 7fecfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:33.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.559+0000 7fecfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:33.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.559+0000 7fecfaffd640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:35:33.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.559+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed041d8470 con 0x7fed040b4d60 2024-03-25T19:35:33.573 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.573+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4200765588 0 0) 0x7fed041d8470 con 0x7fed040b4d60 2024-03-25T19:35:33.573 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.573+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed041d7840 con 0x7fed040b4d60 2024-03-25T19:35:33.573 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.573+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fecec001860 con 0x7fed040b4d60 2024-03-25T19:35:33.573 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.573+0000 7fecfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.573 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.573+0000 7fecfaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:33.573 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.573+0000 7fecfaffd640 10 monclient(hunting): dump: 2024-03-25T19:35:33.573 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:33.573 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:33.573 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.573 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.574 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.574 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:33.574 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:33.574 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:33.574 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:33.574 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:33.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.573+0000 7fecfaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:33.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.582+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2901873023 0 0) 0x7fed0413a8d0 con 0x7fed040b1730 2024-03-25T19:35:33.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.582+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed041d8470 con 0x7fed040b1730 2024-03-25T19:35:33.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.582+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fecf4001b40 con 0x7fed040b1730 2024-03-25T19:35:33.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.582+0000 7fecfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.582+0000 7fecfaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.582+0000 7fecfaffd640 10 monclient(hunting): dump: 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:33.582 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:33.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.585+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3197224499 0 0) 0x7fece4004eb0 con 0x7fed0414be90 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.585+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed0413a8d0 con 0x7fed0414be90 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.585+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fecfc001c90 con 0x7fed0414be90 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.585+0000 7fecfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.585+0000 7fecfaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.585+0000 7fecfaffd640 10 monclient(hunting): dump: 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:33.585 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.599+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 23511335 0 0) 0x7fed041d7840 con 0x7fed040b4d60 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.599+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 >> v1:172.21.15.52:6790/0 conn(0x7fed040b1730 legacy=0x7fed04139390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.599+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 >> v1:172.21.15.52:6789/0 conn(0x7fed0414be90 legacy=0x7fed041e21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fecfaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fecfaffd640 1 monclient: found mon.b 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fecfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fecfaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed041d8600 con 0x7fed040b4d60 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fecfaffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fecfaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fecfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:33.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fed08f9a640 5 monclient: authenticate success, global_id 4600 2024-03-25T19:35:33.601 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fed08f9a640 10 monclient: _renew_subs 2024-03-25T19:35:33.601 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fed08f9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:35:33.601 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fed08f9a640 1 -- 172.21.15.52:0/2100400889 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0413afd0 con 0x7fed040b4d60 2024-03-25T19:35:33.601 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fed08f9a640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:33.601 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fed08f9a640 10 monclient: _renew_subs 2024-03-25T19:35:33.601 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fed08f9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:35:33.601 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fed08f9a640 1 -- 172.21.15.52:0/2100400889 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed04143340 con 0x7fed040b4d60 2024-03-25T19:35:33.601 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.600+0000 7fed08f9a640 1 librados: init done 2024-03-25T19:35:33.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.601+0000 7fed08f9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:33.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.601+0000 7fed08f9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:33.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.601+0000 7fed08f9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:35:33.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.601+0000 7fed08f9a640 1 -- 172.21.15.52:0/2100400889 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed040627b0 con 0x7fed040b4d60 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.905+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecec001d30 con 0x7fed040b4d60 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.905+0000 7fecfaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.905+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fecec003790 con 0x7fed040b4d60 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.905+0000 7fecfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.905+0000 7fecfaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.905+0000 7fecfaffd640 10 monclient: dump: 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:33.905 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:33.906 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.906 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:33.906 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:33.906 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:33.906 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:33.906 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:33.906 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:33.906 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:33.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.905+0000 7fed0159a640 4 set_mon_vals no callback set 2024-03-25T19:35:33.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.906+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fecec019760 con 0x7fed040b4d60 2024-03-25T19:35:33.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.907+0000 7fed0259c640 10 monclient: get_auth_request con 0x7fece406e0b0 auth_method 0 2024-03-25T19:35:33.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.907+0000 7fecfaffd640 10 client.4600.objecter ms_handle_connect 0x7fece406e0b0 2024-03-25T19:35:33.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.911+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fecec0534f0 con 0x7fed040b4d60 2024-03-25T19:35:33.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.911+0000 7fecfaffd640 10 client.4600.objecter ms_dispatch 0x7fed04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:33.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.911+0000 7fecfaffd640 3 client.4600.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:33.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.911+0000 7fecfaffd640 3 client.4600.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:33.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.912+0000 7fecfaffd640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-25T19:35:33.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.912+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fecec04ee30 con 0x7fed040b4d60 2024-03-25T19:35:33.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.912+0000 7fecfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:33.912 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:33.912+0000 7fecfaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:34.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.155+0000 7fed08f9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.154 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.155+0000 7fed08f9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.155+0000 7fed08f9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:35:34.155 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.155+0000 7fed08f9a640 1 -- 172.21.15.52:0/2100400889 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed041d79d0 con 0x7fed040b4d60 2024-03-25T19:35:34.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.248+0000 7fecfaffd640 1 -- 172.21.15.52:0/2100400889 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fecec050f30 con 0x7fed040b4d60 2024-03-25T19:35:34.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.248+0000 7fecfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:34.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.248+0000 7fecfaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:34.248 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:35:34.249 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.251+0000 7fed08f9a640 10 librados: watch_flush enter 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.251+0000 7fed08f9a640 10 librados: watch_flush exit 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.251+0000 7fed08f9a640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-25T19:35:34.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.251+0000 7fed08f9a640 10 client.4600.objecter successfully canceled tick 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.251+0000 7fed08f9a640 1 -- 172.21.15.52:0/2100400889 >> v1:172.21.15.52:6800/3967427147 conn(0x7fece406e0b0 legacy=0x7fece4070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.251+0000 7fed08f9a640 10 monclient: shutdown 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.251+0000 7fed08f9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.251+0000 7fed08f9a640 1 -- 172.21.15.52:0/2100400889 >> v1:172.21.15.179:6789/0 conn(0x7fed040b4d60 legacy=0x7fed041d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.251+0000 7fed08f9a640 1 -- 172.21.15.52:0/2100400889 shutdown_connections 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.251+0000 7fed08f9a640 1 -- 172.21.15.52:0/2100400889 >> 172.21.15.52:0/2100400889 conn(0x7fed040617a0 msgr2=0x7fed040b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.252+0000 7fed08f9a640 1 -- 172.21.15.52:0/2100400889 shutdown_connections 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.252+0000 7fed08f9a640 1 -- 172.21.15.52:0/2100400889 wait complete. 2024-03-25T19:35:34.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:34.252+0000 7fed08f9a640 1 librados: shutdown 2024-03-25T19:35:34.263 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:37.625 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:08:31.224 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.628 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:08:31.227 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.668 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:08:31.267 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.697 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:08:31.239 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.702 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:08:31.244 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.724 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:08:31.323 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.733 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:08:31.332 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.739 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:08:31.281 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:37.763 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:08:31.362 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:41.071 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:41.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.192+0000 7f4911cc8640 10 monclient: get_monmap_and_config 2024-03-25T19:35:41.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.192+0000 7f4911cc8640 10 monclient: build_initial_monmap 2024-03-25T19:35:41.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.192+0000 7f4911cc8640 10 monclient: monmap: 2024-03-25T19:35:41.192 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:41.192 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:41.192 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:35:41.193153+0000 2024-03-25T19:35:41.192 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:35:41.193153+0000 2024-03-25T19:35:41.192 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:41.192 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:41.192 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:41.193 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:35:41.193 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:35:41.193 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:41.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.193+0000 7f4911cc8640 1 Processor -- start 2024-03-25T19:35:41.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.193+0000 7f4911cc8640 1 -- start start 2024-03-25T19:35:41.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.193+0000 7f4911cc8640 10 monclient: init 2024-03-25T19:35:41.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.194+0000 7f4911cc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:41.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.194+0000 7f4911cc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:41.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.194+0000 7f4911cc8640 10 monclient(hunting): picked mon.noname-b con 0x7f490c0a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:41.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.194+0000 7f4911cc8640 10 monclient(hunting): picked mon.noname-c con 0x7f490c13f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:41.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.194+0000 7f4911cc8640 10 monclient(hunting): picked mon.noname-a con 0x7f490c1430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:41.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.194+0000 7f4911cc8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490c142520 con 0x7f490c1430c0 2024-03-25T19:35:41.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.194+0000 7f4911cc8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490c0b28a0 con 0x7f490c13f710 2024-03-25T19:35:41.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.194+0000 7f4911cc8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490c0b2a30 con 0x7f490c0a98f0 2024-03-25T19:35:41.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.194+0000 7f4911cc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:41.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.194+0000 7f4911cc8640 10 monclient(hunting): authenticate will time out at 6389.374058s 2024-03-25T19:35:41.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.196+0000 7f490bfff640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f490c1430c0 0x7f490c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:35742/0 (socket says 172.21.15.52:35742) 2024-03-25T19:35:41.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.196+0000 7f490bfff640 1 -- 172.21.15.52:0/3842649617 learned_addr learned my addr 172.21.15.52:0/3842649617 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:41.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.205+0000 7f490a7fc640 1 -- 172.21.15.52:0/3842649617 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096261723 0 0) 0x7f490c142520 con 0x7f490c1430c0 2024-03-25T19:35:41.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.205+0000 7f490a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.205+0000 7f490a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.205+0000 7f490a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.205+0000 7f490a7fc640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:35:41.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.205+0000 7f490a7fc640 1 -- 172.21.15.52:0/3842649617 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f4003520 con 0x7f490c1430c0 2024-03-25T19:35:41.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f490a7fc640 1 -- 172.21.15.52:0/3842649617 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3987698633 0 0) 0x7f48f4003520 con 0x7f490c1430c0 2024-03-25T19:35:41.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f490a7fc640 1 -- 172.21.15.52:0/3842649617 >> v1:172.21.15.52:6790/0 conn(0x7f490c0a98f0 legacy=0x7f490c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:35:41.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f490a7fc640 1 -- 172.21.15.52:0/3842649617 >> v1:172.21.15.179:6789/0 conn(0x7f490c13f710 legacy=0x7f490c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f490a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:41.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f490a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:35:41.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f490a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:41.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f490a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:35:41.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f490a7fc640 1 -- 172.21.15.52:0/3842649617 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490c0b44c0 con 0x7f490c1430c0 2024-03-25T19:35:41.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f490a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:41.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f490a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:41.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f490a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:41.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f4911cc8640 5 monclient: authenticate success, global_id 4590 2024-03-25T19:35:41.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.211+0000 7f4911cc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.216+0000 7f490a7fc640 1 -- 172.21.15.52:0/3842649617 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4904001a10 con 0x7f490c1430c0 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.216+0000 7f490a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.216+0000 7f490a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.216+0000 7f490a7fc640 10 monclient: dump: 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:41.216 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:41.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.216+0000 7f4911cc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.220+0000 7f490a7fc640 1 -- 172.21.15.52:0/3842649617 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4904001ce0 con 0x7f490c1430c0 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.220+0000 7f490a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.220+0000 7f490a7fc640 1 -- 172.21.15.52:0/3842649617 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4904002070 con 0x7f490c1430c0 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.220+0000 7f4911cc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.220+0000 7f4911cc8640 4 set_mon_vals no callback set 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.220+0000 7f490a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.220+0000 7f490a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.220+0000 7f490a7fc640 10 monclient: dump: 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:41.220 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:41.221 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.221 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:41.221 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:41.221 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:41.221 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:41.221 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:41.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.221+0000 7f4911cc8640 10 monclient: shutdown 2024-03-25T19:35:41.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.221+0000 7f4911cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:41.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.221+0000 7f4911cc8640 1 -- 172.21.15.52:0/3842649617 >> v1:172.21.15.52:6789/0 conn(0x7f490c1430c0 legacy=0x7f490c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.221+0000 7f4911cc8640 1 -- 172.21.15.52:0/3842649617 shutdown_connections 2024-03-25T19:35:41.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.221+0000 7f4911cc8640 1 -- 172.21.15.52:0/3842649617 >> 172.21.15.52:0/3842649617 conn(0x7f490c0b7ea0 msgr2=0x7f490c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.221+0000 7f4911cc8640 1 -- 172.21.15.52:0/3842649617 shutdown_connections 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.221+0000 7f4911cc8640 1 -- 172.21.15.52:0/3842649617 wait complete. 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.222+0000 7f4911cc8640 10 monclient: build_initial_monmap 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.222+0000 7f4911cc8640 10 monclient: monmap: 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:35:41.224 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.222+0000 7f4911cc8640 1 librados: starting msgr at 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.222+0000 7f4911cc8640 1 librados: starting objecter 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.222+0000 7f4911cc8640 1 Processor -- start 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.222+0000 7f4911cc8640 1 -- start start 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.222+0000 7f4911cc8640 1 librados: setting wanted keys 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.222+0000 7f4911cc8640 1 librados: calling monclient init 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.222+0000 7f4911cc8640 10 monclient: init 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.223+0000 7f4911cc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.223+0000 7f4911cc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.223+0000 7f4911cc8640 10 monclient(hunting): picked mon.noname-a con 0x7f490c0a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.223+0000 7f4911cc8640 10 monclient(hunting): picked mon.noname-c con 0x7f490c13f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.223+0000 7f4911cc8640 10 monclient(hunting): picked mon.noname-b con 0x7f490c1430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.223+0000 7f4911cc8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490c0abf60 con 0x7f490c0a98f0 2024-03-25T19:35:41.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.223+0000 7f4911cc8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490c1d77a0 con 0x7f490c1430c0 2024-03-25T19:35:41.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.223+0000 7f4911cc8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f490c1d83d0 con 0x7f490c13f710 2024-03-25T19:35:41.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.223+0000 7f4911cc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:41.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.223+0000 7f4911cc8640 10 monclient(hunting): authenticate will time out at 6389.403861s 2024-03-25T19:35:41.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.234+0000 7f490affd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f490c13f710 0x7f490c1d4820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:35952/0 (socket says 172.21.15.52:35952) 2024-03-25T19:35:41.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.234+0000 7f490affd640 1 -- 172.21.15.52:0/3536194276 learned_addr learned my addr 172.21.15.52:0/3536194276 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:41.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.236+0000 7f48f3fff640 10 client.?.objecter ms_handle_connect 0x7f490c13f710 2024-03-25T19:35:41.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.236+0000 7f48f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.240+0000 7f48f3fff640 10 client.?.objecter ms_handle_connect 0x7f490c1430c0 2024-03-25T19:35:41.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.240+0000 7f48f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.242+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751474517 0 0) 0x7f490c1d83d0 con 0x7f490c13f710 2024-03-25T19:35:41.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.242+0000 7f48f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.242+0000 7f48f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.242+0000 7f48f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.242+0000 7f48f3fff640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:35:41.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.242+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f8004eb0 con 0x7f490c13f710 2024-03-25T19:35:41.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.251+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783279826 0 0) 0x7f490c1d77a0 con 0x7f490c1430c0 2024-03-25T19:35:41.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.251+0000 7f48f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.251+0000 7f48f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.251+0000 7f48f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.251+0000 7f48f3fff640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:35:41.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.251+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f490c1d83d0 con 0x7f490c1430c0 2024-03-25T19:35:41.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.252+0000 7f48f3fff640 10 client.?.objecter ms_handle_connect 0x7f490c0a98f0 2024-03-25T19:35:41.251 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.252+0000 7f48f3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:41.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909907444 0 0) 0x7f490c0abf60 con 0x7f490c0a98f0 2024-03-25T19:35:41.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:41.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:41.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f490c1d77a0 con 0x7f490c0a98f0 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2436154926 0 0) 0x7f490c1d83d0 con 0x7f490c1430c0 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f490c0abf60 con 0x7f490c1430c0 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f49040019d0 con 0x7f490c1430c0 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.263+0000 7f48f3fff640 10 monclient(hunting): dump: 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:41.264 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:41.265 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:41.265 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:41.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.264+0000 7f48f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:41.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.264+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1450203715 0 0) 0x7f48f8004eb0 con 0x7f490c13f710 2024-03-25T19:35:41.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.264+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f490c1d83d0 con 0x7f490c13f710 2024-03-25T19:35:41.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.264+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f49000019d0 con 0x7f490c13f710 2024-03-25T19:35:41.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.264+0000 7f48f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.264+0000 7f48f3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.264+0000 7f48f3fff640 10 monclient(hunting): dump: 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:41.266 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:41.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 917973850 0 0) 0x7f490c1d83d0 con 0x7f490c13f710 2024-03-25T19:35:41.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 >> v1:172.21.15.179:6789/0 conn(0x7f490c1430c0 legacy=0x7f490c1dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 >> v1:172.21.15.52:6789/0 conn(0x7f490c0a98f0 legacy=0x7f490c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f48f3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:41.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f48f3fff640 1 monclient: found mon.c 2024-03-25T19:35:41.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f48f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:41.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f48f3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:41.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490c1d8560 con 0x7f490c13f710 2024-03-25T19:35:41.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f48f3fff640 10 monclient: _finish_auth 0 2024-03-25T19:35:41.269 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f48f3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:41.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f48f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:41.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f4911cc8640 5 monclient: authenticate success, global_id 4589 2024-03-25T19:35:41.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f4911cc8640 10 monclient: _renew_subs 2024-03-25T19:35:41.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f4911cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:41.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f4911cc8640 1 -- 172.21.15.52:0/3536194276 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490c1d41e0 con 0x7f490c13f710 2024-03-25T19:35:41.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f4911cc8640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:41.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f4911cc8640 10 monclient: _renew_subs 2024-03-25T19:35:41.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f4911cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:41.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f4911cc8640 1 -- 172.21.15.52:0/3536194276 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f490c1d5110 con 0x7f490c13f710 2024-03-25T19:35:41.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.269+0000 7f4911cc8640 1 librados: init done 2024-03-25T19:35:41.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.270+0000 7f4911cc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.270+0000 7f4911cc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.270+0000 7f4911cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:41.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.270+0000 7f4911cc8640 1 -- 172.21.15.52:0/3536194276 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490c0b3ef0 con 0x7f490c13f710 2024-03-25T19:35:41.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.282+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4900003610 con 0x7f490c13f710 2024-03-25T19:35:41.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.282+0000 7f48f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:41.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.282+0000 7f4909ffb640 4 set_mon_vals no callback set 2024-03-25T19:35:41.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.305+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f4900003960 con 0x7f490c13f710 2024-03-25T19:35:41.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.305+0000 7f48f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:41.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.305+0000 7f48f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:41.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.305+0000 7f48f3fff640 10 monclient: dump: 2024-03-25T19:35:41.305 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:41.305 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:41.305 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:41.305 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:41.305 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:41.305 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:41.305 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:41.305 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:41.305 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:41.305 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:41.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.305+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f4900003b40 con 0x7f490c13f710 2024-03-25T19:35:41.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.305+0000 7f490b7fe640 10 monclient: get_auth_request con 0x7f48f806dfb0 auth_method 0 2024-03-25T19:35:41.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.306+0000 7f48f3fff640 10 client.4589.objecter ms_handle_connect 0x7f48f806dfb0 2024-03-25T19:35:41.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.313+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f49000548e0 con 0x7f490c13f710 2024-03-25T19:35:41.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.313+0000 7f48f3fff640 10 client.4589.objecter ms_dispatch 0x7f490c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:41.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.313+0000 7f48f3fff640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:41.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.313+0000 7f48f3fff640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:41.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.313+0000 7f48f3fff640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T19:35:41.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.616+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4900050220 con 0x7f490c13f710 2024-03-25T19:35:41.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.616+0000 7f48f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:41.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.616+0000 7f48f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:41.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.854+0000 7f4911cc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:41.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.854+0000 7f4911cc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:41.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.854+0000 7f4911cc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:41.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.854+0000 7f4911cc8640 1 -- 172.21.15.52:0/3536194276 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f490c1d7930 con 0x7f490c13f710 2024-03-25T19:35:41.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.886+0000 7f48f3fff640 1 -- 172.21.15.52:0/3536194276 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f4900052320 con 0x7f490c13f710 2024-03-25T19:35:41.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.886+0000 7f48f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:41.886 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.886+0000 7f48f3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:41.886 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:35:41.886 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 10 librados: watch_flush enter 2024-03-25T19:35:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 10 librados: watch_flush exit 2024-03-25T19:35:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T19:35:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 10 client.4589.objecter successfully canceled tick 2024-03-25T19:35:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 1 -- 172.21.15.52:0/3536194276 >> v1:172.21.15.52:6800/3967427147 conn(0x7f48f806dfb0 legacy=0x7f48f8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 10 monclient: shutdown 2024-03-25T19:35:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 1 -- 172.21.15.52:0/3536194276 >> v1:172.21.15.52:6790/0 conn(0x7f490c13f710 legacy=0x7f490c1d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 1 -- 172.21.15.52:0/3536194276 shutdown_connections 2024-03-25T19:35:41.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 1 -- 172.21.15.52:0/3536194276 >> 172.21.15.52:0/3536194276 conn(0x7f490c0b7ea0 msgr2=0x7f490c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:41.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 1 -- 172.21.15.52:0/3536194276 shutdown_connections 2024-03-25T19:35:41.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 1 -- 172.21.15.52:0/3536194276 wait complete. 2024-03-25T19:35:41.890 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:41.889+0000 7f4911cc8640 1 librados: shutdown 2024-03-25T19:35:41.900 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:48.707 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:48.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.820+0000 7fee7be25640 10 monclient: get_monmap_and_config 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.820+0000 7fee7be25640 10 monclient: build_initial_monmap 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.820+0000 7fee7be25640 10 monclient: monmap: 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:35:48.821525+0000 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:35:48.821525+0000 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:48.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.821+0000 7fee7be25640 1 Processor -- start 2024-03-25T19:35:48.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.821+0000 7fee7be25640 1 -- start start 2024-03-25T19:35:48.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.821+0000 7fee7be25640 10 monclient: init 2024-03-25T19:35:48.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.822+0000 7fee7be25640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:48.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.822+0000 7fee7be25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:48.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.822+0000 7fee7be25640 10 monclient(hunting): picked mon.noname-c con 0x7fee740a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:48.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.822+0000 7fee7be25640 10 monclient(hunting): picked mon.noname-b con 0x7fee7413f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:48.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.822+0000 7fee7be25640 10 monclient(hunting): picked mon.noname-a con 0x7fee741430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:48.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.822+0000 7fee7be25640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee74142520 con 0x7fee741430c0 2024-03-25T19:35:48.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.822+0000 7fee7be25640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740b28a0 con 0x7fee740a98f0 2024-03-25T19:35:48.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.822+0000 7fee7be25640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740b2a30 con 0x7fee7413f710 2024-03-25T19:35:48.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.822+0000 7fee7be25640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:48.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.822+0000 7fee7be25640 10 monclient(hunting): authenticate will time out at 6397.002498s 2024-03-25T19:35:48.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.825+0000 7fee79bc1640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fee740a98f0 0x7fee740a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:46526/0 (socket says 172.21.15.52:46526) 2024-03-25T19:35:48.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.825+0000 7fee79bc1640 1 -- 172.21.15.52:0/401765893 learned_addr learned my addr 172.21.15.52:0/401765893 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:48.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.842+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786576449 0 0) 0x7fee740b28a0 con 0x7fee740a98f0 2024-03-25T19:35:48.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.842+0000 7fee78bbf640 10 monclient(hunting): _negotiate 2024-03-25T19:35:48.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.842+0000 7fee78bbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:48.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.842+0000 7fee78bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:48.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.842+0000 7fee78bbf640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:35:48.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.842+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee60003520 con 0x7fee740a98f0 2024-03-25T19:35:48.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.844+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676702926 0 0) 0x7fee740b2a30 con 0x7fee7413f710 2024-03-25T19:35:48.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.844+0000 7fee78bbf640 10 monclient(hunting): _negotiate 2024-03-25T19:35:48.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.844+0000 7fee78bbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:48.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.844+0000 7fee78bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:48.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.844+0000 7fee78bbf640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:35:48.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.844+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee740b28a0 con 0x7fee7413f710 2024-03-25T19:35:48.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.852+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165095025 0 0) 0x7fee74142520 con 0x7fee741430c0 2024-03-25T19:35:48.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.852+0000 7fee78bbf640 10 monclient(hunting): _negotiate 2024-03-25T19:35:48.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.852+0000 7fee78bbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:48.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.852+0000 7fee78bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:48.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.852+0000 7fee78bbf640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:35:48.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.852+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee740b2a30 con 0x7fee741430c0 2024-03-25T19:35:48.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2392627385 0 0) 0x7fee60003520 con 0x7fee740a98f0 2024-03-25T19:35:48.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 >> v1:172.21.15.52:6790/0 conn(0x7fee7413f710 legacy=0x7fee741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:48.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 >> v1:172.21.15.52:6789/0 conn(0x7fee741430c0 legacy=0x7fee7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:48.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 10 monclient: _finish_hunting 0 2024-03-25T19:35:48.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 1 monclient: found mon.noname-c 2024-03-25T19:35:48.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:48.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:35:48.856 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee740b44c0 con 0x7fee740a98f0 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 10 monclient: _finish_auth 0 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 10 monclient: _check_auth_tickets 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fee70001690 con 0x7fee740a98f0 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee7be25640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee7be25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee78bbf640 10 monclient: dump: 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:48.857 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:48.858 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:48.858 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:48.858 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:48.858 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:48.858 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:48.858 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.855+0000 7fee7be25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:48.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.878+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee700017f0 con 0x7fee740a98f0 2024-03-25T19:35:48.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.878+0000 7fee78bbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:48.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.878+0000 7fee78bbf640 1 -- 172.21.15.52:0/401765893 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fee70001c00 con 0x7fee740a98f0 2024-03-25T19:35:48.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.878+0000 7fee7be25640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:48.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.878+0000 7fee7be25640 4 set_mon_vals no callback set 2024-03-25T19:35:48.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.878+0000 7fee78bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:48.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.878+0000 7fee78bbf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:48.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.878+0000 7fee78bbf640 10 monclient: dump: 2024-03-25T19:35:48.878 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:48.879 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:48.879 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:48.879 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:48.879 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:48.879 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:48.879 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:48.879 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:48.879 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:48.879 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.879+0000 7fee7be25640 10 monclient: shutdown 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.879+0000 7fee7be25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.879+0000 7fee7be25640 1 -- 172.21.15.52:0/401765893 >> v1:172.21.15.179:6789/0 conn(0x7fee740a98f0 legacy=0x7fee740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.879+0000 7fee7be25640 1 -- 172.21.15.52:0/401765893 shutdown_connections 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.879+0000 7fee7be25640 1 -- 172.21.15.52:0/401765893 >> 172.21.15.52:0/401765893 conn(0x7fee740b7ea0 msgr2=0x7fee740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.879+0000 7fee7be25640 1 -- 172.21.15.52:0/401765893 shutdown_connections 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.880+0000 7fee7be25640 1 -- 172.21.15.52:0/401765893 wait complete. 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.880+0000 7fee7be25640 10 monclient: build_initial_monmap 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.880+0000 7fee7be25640 10 monclient: monmap: 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:35:48.881 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.881+0000 7fee7be25640 1 librados: starting msgr at 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.881+0000 7fee7be25640 1 librados: starting objecter 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.881+0000 7fee7be25640 1 Processor -- start 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.881+0000 7fee7be25640 1 -- start start 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.881+0000 7fee7be25640 1 librados: setting wanted keys 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.881+0000 7fee7be25640 1 librados: calling monclient init 2024-03-25T19:35:48.882 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.881+0000 7fee7be25640 10 monclient: init 2024-03-25T19:35:48.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.881+0000 7fee7be25640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:48.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.882+0000 7fee7be25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:48.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.882+0000 7fee7be25640 10 monclient(hunting): picked mon.noname-a con 0x7fee740a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:48.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.882+0000 7fee7be25640 10 monclient(hunting): picked mon.noname-c con 0x7fee7413f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:48.883 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.882+0000 7fee7be25640 10 monclient(hunting): picked mon.noname-b con 0x7fee741430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:48.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.882+0000 7fee7be25640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee741daba0 con 0x7fee740a98f0 2024-03-25T19:35:48.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.882+0000 7fee7be25640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740abac0 con 0x7fee741430c0 2024-03-25T19:35:48.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.882+0000 7fee7be25640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee741db740 con 0x7fee7413f710 2024-03-25T19:35:48.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.882+0000 7fee7be25640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:48.884 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.882+0000 7fee7be25640 10 monclient(hunting): authenticate will time out at 6397.062108s 2024-03-25T19:35:48.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.885+0000 7fee79bc1640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fee740a98f0 0x7fee741db030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:53242/0 (socket says 172.21.15.52:53242) 2024-03-25T19:35:48.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.885+0000 7fee79bc1640 1 -- 172.21.15.52:0/1510625377 learned_addr learned my addr 172.21.15.52:0/1510625377 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:48.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.886+0000 7fee6a7fc640 10 client.?.objecter ms_handle_connect 0x7fee740a98f0 2024-03-25T19:35:48.885 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.886+0000 7fee6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:48.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.899+0000 7fee6a7fc640 10 client.?.objecter ms_handle_connect 0x7fee741430c0 2024-03-25T19:35:48.899 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.899+0000 7fee6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:48.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.901+0000 7fee6a7fc640 10 client.?.objecter ms_handle_connect 0x7fee7413f710 2024-03-25T19:35:48.900 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.901+0000 7fee6a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:48.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.909+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319163321 0 0) 0x7fee741daba0 con 0x7fee740a98f0 2024-03-25T19:35:48.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.909+0000 7fee6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:48.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.909+0000 7fee6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:48.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.909+0000 7fee6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:48.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.909+0000 7fee6a7fc640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:35:48.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.909+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee58004eb0 con 0x7fee740a98f0 2024-03-25T19:35:48.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.914+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258756107 0 0) 0x7fee740abac0 con 0x7fee741430c0 2024-03-25T19:35:48.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.914+0000 7fee6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:48.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.914+0000 7fee6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:48.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.914+0000 7fee6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:48.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.914+0000 7fee6a7fc640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:35:48.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.914+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee741daba0 con 0x7fee741430c0 2024-03-25T19:35:48.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.916+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1287409383 0 0) 0x7fee58004eb0 con 0x7fee740a98f0 2024-03-25T19:35:48.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.916+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee740abac0 con 0x7fee740a98f0 2024-03-25T19:35:48.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.916+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fee70001bc0 con 0x7fee740a98f0 2024-03-25T19:35:48.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.916+0000 7fee6a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:48.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.916+0000 7fee6a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:48.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.916+0000 7fee6a7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:48.916 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:48.916 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:48.916 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:48.917 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:48.917 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:48.917 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:48.917 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:48.917 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:48.917 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:48.917 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:48.917 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.916+0000 7fee6a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:48.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.920+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381336612 0 0) 0x7fee741db740 con 0x7fee7413f710 2024-03-25T19:35:48.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.921+0000 7fee6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:48.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.921+0000 7fee6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:48.920 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.921+0000 7fee6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:48.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.921+0000 7fee6a7fc640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:35:48.921 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.921+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee58004eb0 con 0x7fee7413f710 2024-03-25T19:35:48.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.934+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2138482026 0 0) 0x7fee741daba0 con 0x7fee741430c0 2024-03-25T19:35:48.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.934+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee741db740 con 0x7fee741430c0 2024-03-25T19:35:48.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.934+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fee6c001b50 con 0x7fee741430c0 2024-03-25T19:35:48.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.934+0000 7fee6a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:48.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.934+0000 7fee6a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:48.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.934+0000 7fee6a7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:48.935 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:48.935 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:48.935 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:48.935 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:48.935 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:48.935 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:48.935 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:48.935 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:48.935 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:48.935 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4105400979 0 0) 0x7fee740abac0 con 0x7fee740a98f0 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 >> v1:172.21.15.52:6790/0 conn(0x7fee7413f710 legacy=0x7fee74138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 >> v1:172.21.15.179:6789/0 conn(0x7fee741430c0 legacy=0x7fee741e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee6a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee6a7fc640 1 monclient: found mon.a 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee6a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee741db8d0 con 0x7fee740a98f0 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee6a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee6a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee7be25640 5 monclient: authenticate success, global_id 4599 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee7be25640 10 monclient: _renew_subs 2024-03-25T19:35:48.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee7be25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:35:48.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.935+0000 7fee7be25640 1 -- 172.21.15.52:0/1510625377 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee74138c10 con 0x7fee740a98f0 2024-03-25T19:35:48.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.936+0000 7fee7be25640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:48.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.936+0000 7fee7be25640 10 monclient: _renew_subs 2024-03-25T19:35:48.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.936+0000 7fee7be25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:35:48.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.936+0000 7fee7be25640 1 -- 172.21.15.52:0/1510625377 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee74135190 con 0x7fee740a98f0 2024-03-25T19:35:48.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.936+0000 7fee7be25640 1 librados: init done 2024-03-25T19:35:48.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.937+0000 7fee7be25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:48.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.937+0000 7fee7be25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:48.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.937+0000 7fee7be25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:35:48.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:48.937+0000 7fee7be25640 1 -- 172.21.15.52:0/1510625377 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee740627b0 con 0x7fee740a98f0 2024-03-25T19:35:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.254+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee700011f0 con 0x7fee740a98f0 2024-03-25T19:35:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.254+0000 7fee6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.254+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fee70004c30 con 0x7fee740a98f0 2024-03-25T19:35:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.254+0000 7fee6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.254+0000 7fee6a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:49.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.254+0000 7fee6a7fc640 10 monclient: dump: 2024-03-25T19:35:49.256 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:49.256 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:49.256 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:49.256 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:49.256 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:49.256 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:49.256 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:49.256 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:49.256 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:49.256 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:49.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.254+0000 7fee6bfff640 4 set_mon_vals no callback set 2024-03-25T19:35:49.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.255+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fee70004e50 con 0x7fee740a98f0 2024-03-25T19:35:49.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.256+0000 7fee793c0640 10 monclient: get_auth_request con 0x7fee5806df60 auth_method 0 2024-03-25T19:35:49.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.256+0000 7fee6a7fc640 10 client.4599.objecter ms_handle_connect 0x7fee5806df60 2024-03-25T19:35:49.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.261+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fee70054960 con 0x7fee740a98f0 2024-03-25T19:35:49.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.261+0000 7fee6a7fc640 10 client.4599.objecter ms_dispatch 0x7fee74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:49.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.261+0000 7fee6a7fc640 3 client.4599.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:49.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.261+0000 7fee6a7fc640 3 client.4599.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:49.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.262+0000 7fee6a7fc640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-25T19:35:49.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.262+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee700502a0 con 0x7fee740a98f0 2024-03-25T19:35:49.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.262+0000 7fee6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:49.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.262+0000 7fee6a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:49.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.494+0000 7fee7be25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:49.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.494+0000 7fee7be25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:49.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.494+0000 7fee7be25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:35:49.495 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.494+0000 7fee7be25640 1 -- 172.21.15.52:0/1510625377 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee74074cc0 con 0x7fee740a98f0 2024-03-25T19:35:49.541 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.540+0000 7fee6a7fc640 1 -- 172.21.15.52:0/1510625377 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fee700523a0 con 0x7fee740a98f0 2024-03-25T19:35:49.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.541+0000 7fee6a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:49.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.541+0000 7fee6a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:49.583 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:35:49.584 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:49.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 10 librados: watch_flush enter 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 10 librados: watch_flush exit 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 10 client.4599.objecter successfully canceled tick 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 1 -- 172.21.15.52:0/1510625377 >> v1:172.21.15.52:6800/3967427147 conn(0x7fee5806df60 legacy=0x7fee58070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 10 monclient: shutdown 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 1 -- 172.21.15.52:0/1510625377 >> v1:172.21.15.52:6789/0 conn(0x7fee740a98f0 legacy=0x7fee741db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 1 -- 172.21.15.52:0/1510625377 shutdown_connections 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 1 -- 172.21.15.52:0/1510625377 >> 172.21.15.52:0/1510625377 conn(0x7fee740b7ea0 msgr2=0x7fee74074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 1 -- 172.21.15.52:0/1510625377 shutdown_connections 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 1 -- 172.21.15.52:0/1510625377 wait complete. 2024-03-25T19:35:49.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:49.543+0000 7fee7be25640 1 librados: shutdown 2024-03-25T19:35:49.587 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:56.393 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:56.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.507+0000 7f46e3c86640 10 monclient: get_monmap_and_config 2024-03-25T19:35:56.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.507+0000 7f46e3c86640 10 monclient: build_initial_monmap 2024-03-25T19:35:56.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.507+0000 7f46e3c86640 10 monclient: monmap: 2024-03-25T19:35:56.508 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:56.508 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:56.508 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:35:56.509451+0000 2024-03-25T19:35:56.508 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:35:56.509451+0000 2024-03-25T19:35:56.508 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:56.509 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:56.509 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:56.509 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:35:56.509 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:35:56.509 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:56.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.508+0000 7f46e3c86640 1 Processor -- start 2024-03-25T19:35:56.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.508+0000 7f46e3c86640 1 -- start start 2024-03-25T19:35:56.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.508+0000 7f46e3c86640 10 monclient: init 2024-03-25T19:35:56.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.509+0000 7f46e3c86640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:56.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.509+0000 7f46e3c86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:56.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.509+0000 7f46e3c86640 10 monclient(hunting): picked mon.noname-a con 0x7f46dc0a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:56.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.509+0000 7f46e3c86640 10 monclient(hunting): picked mon.noname-c con 0x7f46dc13f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:56.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.509+0000 7f46e3c86640 10 monclient(hunting): picked mon.noname-b con 0x7f46dc1430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:56.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.509+0000 7f46e3c86640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc142520 con 0x7f46dc0a98f0 2024-03-25T19:35:56.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.509+0000 7f46e3c86640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc0b28a0 con 0x7f46dc13f710 2024-03-25T19:35:56.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.510+0000 7f46e3c86640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc0b2a30 con 0x7f46dc1430c0 2024-03-25T19:35:56.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.510+0000 7f46e3c86640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:56.511 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.510+0000 7f46e3c86640 10 monclient(hunting): authenticate will time out at 6404.690967s 2024-03-25T19:35:56.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.514+0000 7f46e1221640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f46dc13f710 0x7f46dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:54334/0 (socket says 172.21.15.52:54334) 2024-03-25T19:35:56.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.514+0000 7f46e1221640 1 -- 172.21.15.52:0/463833081 learned_addr learned my addr 172.21.15.52:0/463833081 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:56.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.526+0000 7f46e0a20640 1 -- 172.21.15.52:0/463833081 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197958998 0 0) 0x7f46dc0b28a0 con 0x7f46dc13f710 2024-03-25T19:35:56.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.526+0000 7f46e0a20640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.526+0000 7f46e0a20640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.526+0000 7f46e0a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.526+0000 7f46e0a20640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:35:56.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.526+0000 7f46e0a20640 1 -- 172.21.15.52:0/463833081 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c8003520 con 0x7f46dc13f710 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 1 -- 172.21.15.52:0/463833081 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 15961054 0 0) 0x7f46c8003520 con 0x7f46dc13f710 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 1 -- 172.21.15.52:0/463833081 >> v1:172.21.15.52:6790/0 conn(0x7f46dc1430c0 legacy=0x7f46dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 1 -- 172.21.15.52:0/463833081 >> v1:172.21.15.52:6789/0 conn(0x7f46dc0a98f0 legacy=0x7f46dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 10 monclient: _finish_hunting 0 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 1 monclient: found mon.noname-c 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 1 -- 172.21.15.52:0/463833081 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc0b44c0 con 0x7f46dc13f710 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 10 monclient: _finish_auth 0 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 10 monclient: _check_auth_tickets 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 1 -- 172.21.15.52:0/463833081 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46cc001690 con 0x7f46dc13f710 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e0a20640 10 monclient: dump: 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e3c86640 5 monclient: authenticate success, global_id 4615 2024-03-25T19:35:56.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.533+0000 7f46e3c86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:56.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.546+0000 7f46e0a20640 1 -- 172.21.15.52:0/463833081 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46cc0017f0 con 0x7f46dc13f710 2024-03-25T19:35:56.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.546+0000 7f46e0a20640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:56.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.546+0000 7f46e0a20640 1 -- 172.21.15.52:0/463833081 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46cc001c00 con 0x7f46dc13f710 2024-03-25T19:35:56.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.546+0000 7f46e3c86640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:56.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.546+0000 7f46e3c86640 4 set_mon_vals no callback set 2024-03-25T19:35:56.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.546+0000 7f46e0a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.546+0000 7f46e0a20640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:56.547 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.546+0000 7f46e0a20640 10 monclient: dump: 2024-03-25T19:35:56.547 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:56.547 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:56.548 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:56.548 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:56.548 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.548 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:56.548 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:56.548 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:56.548 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:56.548 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:56.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.547+0000 7f46e3c86640 10 monclient: shutdown 2024-03-25T19:35:56.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.547+0000 7f46e3c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:56.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.547+0000 7f46e3c86640 1 -- 172.21.15.52:0/463833081 >> v1:172.21.15.179:6789/0 conn(0x7f46dc13f710 legacy=0x7f46dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.547+0000 7f46e3c86640 1 -- 172.21.15.52:0/463833081 shutdown_connections 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.547+0000 7f46e3c86640 1 -- 172.21.15.52:0/463833081 >> 172.21.15.52:0/463833081 conn(0x7f46dc0b7ea0 msgr2=0x7f46dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.547+0000 7f46e3c86640 1 -- 172.21.15.52:0/463833081 shutdown_connections 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.547+0000 7f46e3c86640 1 -- 172.21.15.52:0/463833081 wait complete. 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.547+0000 7f46e3c86640 10 monclient: build_initial_monmap 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.547+0000 7f46e3c86640 10 monclient: monmap: 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:35:56.551 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.548+0000 7f46e3c86640 1 librados: starting msgr at 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.548+0000 7f46e3c86640 1 librados: starting objecter 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.548+0000 7f46e3c86640 1 Processor -- start 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.548+0000 7f46e3c86640 1 -- start start 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.548+0000 7f46e3c86640 1 librados: setting wanted keys 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.548+0000 7f46e3c86640 1 librados: calling monclient init 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.548+0000 7f46e3c86640 10 monclient: init 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.549+0000 7f46e3c86640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.549+0000 7f46e3c86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.549+0000 7f46e3c86640 10 monclient(hunting): picked mon.noname-b con 0x7f46dc0a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:35:56.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.549+0000 7f46e3c86640 10 monclient(hunting): picked mon.noname-a con 0x7f46dc13f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:35:56.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.549+0000 7f46e3c86640 10 monclient(hunting): picked mon.noname-c con 0x7f46dc1430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:35:56.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.549+0000 7f46e3c86640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc137df0 con 0x7f46dc13f710 2024-03-25T19:35:56.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.549+0000 7f46e3c86640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc1337a0 con 0x7f46dc0a98f0 2024-03-25T19:35:56.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.549+0000 7f46e3c86640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46dc133930 con 0x7f46dc1430c0 2024-03-25T19:35:56.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.549+0000 7f46e3c86640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:56.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.549+0000 7f46e3c86640 10 monclient(hunting): authenticate will time out at 6404.730246s 2024-03-25T19:35:56.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.554+0000 7f46e2223640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f46dc1430c0 0x7f46dc133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:49634/0 (socket says 172.21.15.52:49634) 2024-03-25T19:35:56.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.554+0000 7f46e2223640 1 -- 172.21.15.52:0/3266126631 learned_addr learned my addr 172.21.15.52:0/3266126631 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:35:56.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.559+0000 7f46d27fc640 10 client.?.objecter ms_handle_connect 0x7f46dc1430c0 2024-03-25T19:35:56.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.559+0000 7f46d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:56.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.562+0000 7f46d27fc640 10 client.?.objecter ms_handle_connect 0x7f46dc13f710 2024-03-25T19:35:56.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.562+0000 7f46d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:56.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.567+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998440347 0 0) 0x7f46dc133930 con 0x7f46dc1430c0 2024-03-25T19:35:56.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.567+0000 7f46d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.567+0000 7f46d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.567+0000 7f46d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.567+0000 7f46d27fc640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:35:56.568 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.567+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c0004eb0 con 0x7f46dc1430c0 2024-03-25T19:35:56.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.569+0000 7f46d27fc640 10 client.?.objecter ms_handle_connect 0x7f46dc0a98f0 2024-03-25T19:35:56.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.569+0000 7f46d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:56.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.575+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930317156 0 0) 0x7f46dc137df0 con 0x7f46dc13f710 2024-03-25T19:35:56.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.575+0000 7f46d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.575+0000 7f46d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.575+0000 7f46d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.575+0000 7f46d27fc640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:35:56.575 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.575+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc133930 con 0x7f46dc13f710 2024-03-25T19:35:56.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.585+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130947040 0 0) 0x7f46dc1337a0 con 0x7f46dc0a98f0 2024-03-25T19:35:56.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.585+0000 7f46d27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:56.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.585+0000 7f46d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:56.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.585+0000 7f46d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:56.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.585+0000 7f46d27fc640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:35:56.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.585+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46dc137df0 con 0x7f46dc0a98f0 2024-03-25T19:35:56.590 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.590+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1310501034 0 0) 0x7f46c0004eb0 con 0x7f46dc1430c0 2024-03-25T19:35:56.590 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.590+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46dc1337a0 con 0x7f46dc1430c0 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.590+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46d4001c90 con 0x7f46dc1430c0 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.590+0000 7f46d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.590+0000 7f46d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.590+0000 7f46d27fc640 10 monclient(hunting): dump: 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:56.591 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:56.592 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:56.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.590+0000 7f46d27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:56.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.591+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1993798809 0 0) 0x7f46dc133930 con 0x7f46dc13f710 2024-03-25T19:35:56.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.591+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46c0004eb0 con 0x7f46dc13f710 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.591+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46cc001b40 con 0x7f46dc13f710 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.591+0000 7f46d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.591+0000 7f46d27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.591+0000 7f46d27fc640 10 monclient(hunting): dump: 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:56.593 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.600+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2806471870 0 0) 0x7f46dc1337a0 con 0x7f46dc1430c0 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.600+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 >> v1:172.21.15.179:6789/0 conn(0x7f46dc0a98f0 legacy=0x7f46dc1392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.600+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 >> v1:172.21.15.52:6789/0 conn(0x7f46dc13f710 legacy=0x7f46dc1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.600+0000 7f46d27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.600+0000 7f46d27fc640 1 monclient: found mon.c 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.600+0000 7f46d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.600+0000 7f46d27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc1367d0 con 0x7f46dc1430c0 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46d27fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46d27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46e3c86640 5 monclient: authenticate success, global_id 4601 2024-03-25T19:35:56.602 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46e3c86640 10 monclient: _renew_subs 2024-03-25T19:35:56.603 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46e3c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:56.603 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46e3c86640 1 -- 172.21.15.52:0/3266126631 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46dc1369e0 con 0x7f46dc1430c0 2024-03-25T19:35:56.603 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46e3c86640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:56.603 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46e3c86640 10 monclient: _renew_subs 2024-03-25T19:35:56.603 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46e3c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:56.603 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46e3c86640 1 -- 172.21.15.52:0/3266126631 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc13b640 con 0x7f46dc1430c0 2024-03-25T19:35:56.603 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.601+0000 7f46e3c86640 1 librados: init done 2024-03-25T19:35:56.604 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.602+0000 7f46e3c86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:56.604 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.602+0000 7f46e3c86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:56.604 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.602+0000 7f46e3c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:56.604 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.602+0000 7f46e3c86640 1 -- 172.21.15.52:0/3266126631 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46dc0627b0 con 0x7f46dc1430c0 2024-03-25T19:35:56.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.921+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46d4003990 con 0x7f46dc1430c0 2024-03-25T19:35:56.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.921+0000 7f46d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:56.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.921+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f46d4003d60 con 0x7f46dc1430c0 2024-03-25T19:35:56.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.921+0000 7f46d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:56.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.921+0000 7f46d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:56.922 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.921+0000 7f46d27fc640 10 monclient: dump: 2024-03-25T19:35:56.922 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:35:56.923 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:35:56.923 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:56.923 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:35:56.923 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:56.923 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:35:56.923 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:35:56.923 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:35:56.923 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:35:56.923 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:35:56.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.921+0000 7f46d3fff640 4 set_mon_vals no callback set 2024-03-25T19:35:56.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.922+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f46d4003f80 con 0x7f46dc1430c0 2024-03-25T19:35:56.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.923+0000 7f46e1a22640 10 monclient: get_auth_request con 0x7f46c006df70 auth_method 0 2024-03-25T19:35:56.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.923+0000 7f46d27fc640 10 client.4601.objecter ms_handle_connect 0x7f46c006df70 2024-03-25T19:35:56.929 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.929+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f46d4054b00 con 0x7f46dc1430c0 2024-03-25T19:35:56.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.929+0000 7f46d27fc640 10 client.4601.objecter ms_dispatch 0x7f46dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:56.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.929+0000 7f46d27fc640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:56.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.929+0000 7f46d27fc640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:56.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.929+0000 7f46d27fc640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-25T19:35:56.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.929+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46d4050610 con 0x7f46dc1430c0 2024-03-25T19:35:56.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.929+0000 7f46d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:56.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:56.929+0000 7f46d27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.143+0000 7f46e3c86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.143+0000 7f46e3c86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.143+0000 7f46e3c86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:35:57.144 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.143+0000 7f46e3c86640 1 -- 172.21.15.52:0/3266126631 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46dc074cc0 con 0x7f46dc1430c0 2024-03-25T19:35:57.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.254+0000 7f46d27fc640 1 -- 172.21.15.52:0/3266126631 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f46d4052540 con 0x7f46dc1430c0 2024-03-25T19:35:57.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.254+0000 7f46d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:57.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.254+0000 7f46d27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:57.255 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:35:57.255 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:57.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 10 librados: watch_flush enter 2024-03-25T19:35:57.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 10 librados: watch_flush exit 2024-03-25T19:35:57.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-25T19:35:57.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 10 client.4601.objecter successfully canceled tick 2024-03-25T19:35:57.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 1 -- 172.21.15.52:0/3266126631 >> v1:172.21.15.52:6800/3967427147 conn(0x7f46c006df70 legacy=0x7f46c0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 10 monclient: shutdown 2024-03-25T19:35:57.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:57.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 1 -- 172.21.15.52:0/3266126631 >> v1:172.21.15.52:6790/0 conn(0x7f46dc1430c0 legacy=0x7f46dc133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:57.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 1 -- 172.21.15.52:0/3266126631 shutdown_connections 2024-03-25T19:35:57.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 1 -- 172.21.15.52:0/3266126631 >> 172.21.15.52:0/3266126631 conn(0x7f46dc0b7ea0 msgr2=0x7f46dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:57.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 1 -- 172.21.15.52:0/3266126631 shutdown_connections 2024-03-25T19:35:57.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 1 -- 172.21.15.52:0/3266126631 wait complete. 2024-03-25T19:35:57.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:35:57.257+0000 7f46e3c86640 1 librados: shutdown 2024-03-25T19:35:57.269 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:57.629 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:08:51.228 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.679 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:08:51.278 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.702 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:08:51.301 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.741 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:08:51.282 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.747 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:08:51.346 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.752 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:08:51.294 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.759 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:08:51.357 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.786 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:08:51.327 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:57.792 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:08:51.391 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:04.076 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:04.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.177+0000 7f7eccf82640 10 monclient: get_monmap_and_config 2024-03-25T19:36:04.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.177+0000 7f7eccf82640 10 monclient: build_initial_monmap 2024-03-25T19:36:04.178 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.177+0000 7f7eccf82640 10 monclient: monmap: 2024-03-25T19:36:04.178 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:04.178 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:04.178 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:36:04.179221+0000 2024-03-25T19:36:04.178 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:36:04.179221+0000 2024-03-25T19:36:04.178 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:04.178 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:04.179 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:04.179 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:36:04.179 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:36:04.179 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:04.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.178+0000 7f7eccf82640 1 Processor -- start 2024-03-25T19:36:04.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.178+0000 7f7eccf82640 1 -- start start 2024-03-25T19:36:04.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.178+0000 7f7eccf82640 10 monclient: init 2024-03-25T19:36:04.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.179+0000 7f7eccf82640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:04.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.179+0000 7f7eccf82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:04.180 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.179+0000 7f7eccf82640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec80b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:04.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.179+0000 7f7eccf82640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec80b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:04.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.179+0000 7f7eccf82640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec814be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:04.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.179+0000 7f7eccf82640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80af940 con 0x7f7ec814be90 2024-03-25T19:36:04.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.179+0000 7f7eccf82640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80b0a70 con 0x7f7ec80b4d60 2024-03-25T19:36:04.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.179+0000 7f7eccf82640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80b0d50 con 0x7f7ec80b1730 2024-03-25T19:36:04.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.179+0000 7f7eccf82640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:04.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.179+0000 7f7eccf82640 10 monclient(hunting): authenticate will time out at 6412.360086s 2024-03-25T19:36:04.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.193+0000 7f7ec6d9d640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f7ec814be90 0x7f7ec814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:48942/0 (socket says 172.21.15.52:48942) 2024-03-25T19:36:04.194 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.193+0000 7f7ec6d9d640 1 -- 172.21.15.52:0/1948853114 learned_addr learned my addr 172.21.15.52:0/1948853114 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:04.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.201+0000 7f7ec6d9d640 1 -- 172.21.15.52:0/1948853114 >> v1:172.21.15.52:6789/0 conn(0x7f7ec814be90 legacy=0x7f7ec814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T19:36:04.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.202+0000 7f7ec6d9d640 1 -- 172.21.15.52:0/1948853114 >> v1:172.21.15.52:6789/0 conn(0x7f7ec814be90 legacy=0x7f7ec814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:36:04.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.202+0000 7f7ec6d9d640 1 --1- 172.21.15.52:0/1948853114 >> v1:172.21.15.52:6789/0 conn(0x7f7ec814be90 0x7f7ec814e280 :-1 s=OPENED pgs=62441 cs=1 l=1).handle_message read tag failed 2024-03-25T19:36:04.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.202+0000 7f7ec6d9d640 1 --1- 172.21.15.52:0/1948853114 >> v1:172.21.15.52:6789/0 conn(0x7f7ec814be90 0x7f7ec814e280 :-1 s=OPENED pgs=62441 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:36:04.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.202+0000 7f7ec5d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.52:6789/0 2024-03-25T19:36:04.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.213+0000 7f7ec5d9b640 1 -- 172.21.15.52:0/1948853114 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517929827 0 0) 0x7f7ec80b0d50 con 0x7f7ec80b1730 2024-03-25T19:36:04.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.213+0000 7f7ec5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.213+0000 7f7ec5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.213+0000 7f7ec5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.213+0000 7f7ec5d9b640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:36:04.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.213+0000 7f7ec5d9b640 1 -- 172.21.15.52:0/1948853114 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb4003520 con 0x7f7ec80b1730 2024-03-25T19:36:04.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.217+0000 7f7ec5d9b640 1 -- 172.21.15.52:0/1948853114 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754768066 0 0) 0x7f7ec80b0a70 con 0x7f7ec80b4d60 2024-03-25T19:36:04.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.217+0000 7f7ec5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.217+0000 7f7ec5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.217+0000 7f7ec5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.217+0000 7f7ec5d9b640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:36:04.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.217+0000 7f7ec5d9b640 1 -- 172.21.15.52:0/1948853114 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec80b0d50 con 0x7f7ec80b4d60 2024-03-25T19:36:04.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 1 -- 172.21.15.52:0/1948853114 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3737605145 0 0) 0x7f7ec80b0d50 con 0x7f7ec80b4d60 2024-03-25T19:36:04.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 1 -- 172.21.15.52:0/1948853114 >> v1:172.21.15.52:6790/0 conn(0x7f7ec80b1730 legacy=0x7f7ec80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 1 -- 172.21.15.52:0/1948853114 >> v1:172.21.15.52:6789/0 conn(0x7f7ec814be90 legacy=0x7f7ec814e280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:36:04.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:36:04.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 1 monclient: found mon.noname-c 2024-03-25T19:36:04.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:04.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:36:04.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 1 -- 172.21.15.52:0/1948853114 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec80b1030 con 0x7f7ec80b4d60 2024-03-25T19:36:04.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 1 -- 172.21.15.52:0/1948853114 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7eb8001630 con 0x7f7ec80b4d60 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7eccf82640 5 monclient: authenticate success, global_id 4621 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7eccf82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7ec5d9b640 10 monclient: dump: 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.234 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:04.235 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:04.235 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:04.235 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:04.235 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:04.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.232+0000 7f7eccf82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:04.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.252+0000 7f7ec5d9b640 1 -- 172.21.15.52:0/1948853114 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7eb8001790 con 0x7f7ec80b4d60 2024-03-25T19:36:04.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.252+0000 7f7ec5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:04.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.252+0000 7f7ec5d9b640 1 -- 172.21.15.52:0/1948853114 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7eb8001be0 con 0x7f7ec80b4d60 2024-03-25T19:36:04.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.252+0000 7f7ec5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.252+0000 7f7ec5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:04.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.252+0000 7f7ec5d9b640 10 monclient: dump: 2024-03-25T19:36:04.253 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:04.253 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:04.253 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:04.254 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:04.254 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.254 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:04.254 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:04.254 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:04.254 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:04.254 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:04.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.252+0000 7f7eccf82640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:04.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.252+0000 7f7eccf82640 4 set_mon_vals no callback set 2024-03-25T19:36:04.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.253+0000 7f7eccf82640 10 monclient: shutdown 2024-03-25T19:36:04.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.253+0000 7f7eccf82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:04.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.253+0000 7f7eccf82640 1 -- 172.21.15.52:0/1948853114 >> v1:172.21.15.179:6789/0 conn(0x7f7ec80b4d60 legacy=0x7f7ec813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.253+0000 7f7eccf82640 1 -- 172.21.15.52:0/1948853114 shutdown_connections 2024-03-25T19:36:04.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.253+0000 7f7eccf82640 1 -- 172.21.15.52:0/1948853114 >> 172.21.15.52:0/1948853114 conn(0x7f7ec80617a0 msgr2=0x7f7ec8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:04.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.253+0000 7f7eccf82640 1 -- 172.21.15.52:0/1948853114 shutdown_connections 2024-03-25T19:36:04.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.253+0000 7f7eccf82640 1 -- 172.21.15.52:0/1948853114 wait complete. 2024-03-25T19:36:04.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.255+0000 7f7eccf82640 10 monclient: build_initial_monmap 2024-03-25T19:36:04.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.255+0000 7f7eccf82640 10 monclient: monmap: 2024-03-25T19:36:04.256 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:04.256 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:04.256 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:36:04.256 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:36:04.256 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:04.256 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:36:04.256 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:04.256 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:36:04.256 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:36:04.256 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:04.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.255+0000 7f7eccf82640 1 librados: starting msgr at 2024-03-25T19:36:04.257 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.255+0000 7f7eccf82640 1 librados: starting objecter 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.255+0000 7f7eccf82640 1 Processor -- start 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.255+0000 7f7eccf82640 1 -- start start 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.255+0000 7f7eccf82640 1 librados: setting wanted keys 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 1 librados: calling monclient init 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 10 monclient: init 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec80b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec80b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec814be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec81419e0 con 0x7f7ec80b1730 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec813b8b0 con 0x7f7ec814be90 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec813ad00 con 0x7f7ec80b4d60 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:04.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.256+0000 7f7eccf82640 10 monclient(hunting): authenticate will time out at 6412.437577s 2024-03-25T19:36:04.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.259+0000 7f7ec659c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f7ec80b1730 0x7f7ec813a0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:48996/0 (socket says 172.21.15.52:48996) 2024-03-25T19:36:04.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.260+0000 7f7ec659c640 1 -- 172.21.15.52:0/3299156180 learned_addr learned my addr 172.21.15.52:0/3299156180 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:04.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.260+0000 7f7ebf7fe640 10 client.?.objecter ms_handle_connect 0x7f7ec80b1730 2024-03-25T19:36:04.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.260+0000 7f7ebf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.269+0000 7f7ebf7fe640 10 client.?.objecter ms_handle_connect 0x7f7ec814be90 2024-03-25T19:36:04.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.269+0000 7f7ebf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.270+0000 7f7ebf7fe640 10 client.?.objecter ms_handle_connect 0x7f7ec80b4d60 2024-03-25T19:36:04.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.270+0000 7f7ebf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:04.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.274+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146878677 0 0) 0x7f7ec81419e0 con 0x7f7ec80b1730 2024-03-25T19:36:04.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.274+0000 7f7ebf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.274+0000 7f7ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.274+0000 7f7ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.274+0000 7f7ebf7fe640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:36:04.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.274+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea8004eb0 con 0x7f7ec80b1730 2024-03-25T19:36:04.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.276+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266513152 0 0) 0x7f7ec813ad00 con 0x7f7ec80b4d60 2024-03-25T19:36:04.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.276+0000 7f7ebf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.276+0000 7f7ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.276+0000 7f7ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.276+0000 7f7ebf7fe640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:36:04.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.276+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec81419e0 con 0x7f7ec80b4d60 2024-03-25T19:36:04.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.288+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209985726 0 0) 0x7f7ea8004eb0 con 0x7f7ec80b1730 2024-03-25T19:36:04.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.288+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ec813ad00 con 0x7f7ec80b1730 2024-03-25T19:36:04.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.288+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7eb80015f0 con 0x7f7ec80b1730 2024-03-25T19:36:04.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.288+0000 7f7ebf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.289 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.288+0000 7f7ebf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.288+0000 7f7ebf7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:04.290 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.288+0000 7f7ebf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:04.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.290+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 639474534 0 0) 0x7f7ec81419e0 con 0x7f7ec80b4d60 2024-03-25T19:36:04.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.290+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ea8004eb0 con 0x7f7ec80b4d60 2024-03-25T19:36:04.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.290+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7eb0001950 con 0x7f7ec80b4d60 2024-03-25T19:36:04.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.290+0000 7f7ebf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.290+0000 7f7ebf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:04.291 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.290+0000 7f7ebf7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:04.291 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:04.291 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:04.292 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:04.292 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:04.292 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.292 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:04.292 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:04.292 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:04.292 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:04.292 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:04.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.296+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203632915 0 0) 0x7f7ec813b8b0 con 0x7f7ec814be90 2024-03-25T19:36:04.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.296+0000 7f7ebf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:04.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.296+0000 7f7ebf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:04.296 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.296+0000 7f7ebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:04.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.296+0000 7f7ebf7fe640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:36:04.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.296+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec81419e0 con 0x7f7ec814be90 2024-03-25T19:36:04.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3557108336 0 0) 0x7f7ec813ad00 con 0x7f7ec80b1730 2024-03-25T19:36:04.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 >> v1:172.21.15.52:6790/0 conn(0x7f7ec80b4d60 legacy=0x7f7ec81365e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 >> v1:172.21.15.179:6789/0 conn(0x7f7ec814be90 legacy=0x7f7ec81dbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7ebf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:04.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7ebf7fe640 1 monclient: found mon.a 2024-03-25T19:36:04.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7ebf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:04.299 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7ebf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec81432f0 con 0x7f7ec80b1730 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7ebf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7ebf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7ebf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7eccf82640 5 monclient: authenticate success, global_id 4611 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7eccf82640 10 monclient: _renew_subs 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7eccf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7eccf82640 1 -- 172.21.15.52:0/3299156180 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec813bb00 con 0x7f7ec80b1730 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7eccf82640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7eccf82640 10 monclient: _renew_subs 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7eccf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7eccf82640 1 -- 172.21.15.52:0/3299156180 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec813aa80 con 0x7f7ec80b1730 2024-03-25T19:36:04.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.298+0000 7f7eccf82640 1 librados: init done 2024-03-25T19:36:04.301 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.299+0000 7f7eccf82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:04.301 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.299+0000 7f7eccf82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:04.301 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.299+0000 7f7eccf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:36:04.301 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.299+0000 7f7eccf82640 1 -- 172.21.15.52:0/3299156180 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec80afbd0 con 0x7f7ec80b1730 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.611+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7eb80038a0 con 0x7f7ec80b1730 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.611+0000 7f7ebf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:04.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.611+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7eb8003bf0 con 0x7f7ec80b1730 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.611+0000 7f7ebf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.611+0000 7f7ebf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.611+0000 7f7ebf7fe640 10 monclient: dump: 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:04.612 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.611+0000 7f7ec559a640 4 set_mon_vals no callback set 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.611+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f7eb8003e10 con 0x7f7ec80b1730 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.612+0000 7f7ebdd9b640 10 monclient: get_auth_request con 0x7f7ea806dec0 auth_method 0 2024-03-25T19:36:04.613 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.612+0000 7f7ebf7fe640 10 client.4611.objecter ms_handle_connect 0x7f7ea806dec0 2024-03-25T19:36:04.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.619+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f7eb8054920 con 0x7f7ec80b1730 2024-03-25T19:36:04.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.619+0000 7f7ebf7fe640 10 client.4611.objecter ms_dispatch 0x7f7ec8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:04.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.619+0000 7f7ebf7fe640 3 client.4611.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:04.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.619+0000 7f7ebf7fe640 3 client.4611.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:04.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.619+0000 7f7ebf7fe640 20 client.4611.objecter dump_active .. 0 homeless 2024-03-25T19:36:04.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.619+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7eb8050430 con 0x7f7ec80b1730 2024-03-25T19:36:04.620 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.619+0000 7f7ebf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:04.621 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.619+0000 7f7ebf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:04.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.880+0000 7f7eccf82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:04.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.880+0000 7f7eccf82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:04.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.880+0000 7f7eccf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:36:04.881 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.880+0000 7f7eccf82640 1 -- 172.21.15.52:0/3299156180 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ec80627b0 con 0x7f7ec80b1730 2024-03-25T19:36:04.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.923+0000 7f7ebf7fe640 1 -- 172.21.15.52:0/3299156180 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f7eb8052360 con 0x7f7ec80b1730 2024-03-25T19:36:04.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.923+0000 7f7ebf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:04.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.923+0000 7f7ebf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:04.947 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:36:04.947 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:04.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.925+0000 7f7eccf82640 10 librados: watch_flush enter 2024-03-25T19:36:04.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 10 librados: watch_flush exit 2024-03-25T19:36:04.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 20 client.4611.objecter shutdown clearing up homeless session... 2024-03-25T19:36:04.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 10 client.4611.objecter successfully canceled tick 2024-03-25T19:36:04.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 1 -- 172.21.15.52:0/3299156180 >> v1:172.21.15.52:6800/3967427147 conn(0x7f7ea806dec0 legacy=0x7f7ea8070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 10 monclient: shutdown 2024-03-25T19:36:04.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:04.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 1 -- 172.21.15.52:0/3299156180 >> v1:172.21.15.52:6789/0 conn(0x7f7ec80b1730 legacy=0x7f7ec813a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:04.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 1 -- 172.21.15.52:0/3299156180 shutdown_connections 2024-03-25T19:36:04.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 1 -- 172.21.15.52:0/3299156180 >> 172.21.15.52:0/3299156180 conn(0x7f7ec80617a0 msgr2=0x7f7ec813c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:04.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 1 -- 172.21.15.52:0/3299156180 shutdown_connections 2024-03-25T19:36:04.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 1 -- 172.21.15.52:0/3299156180 wait complete. 2024-03-25T19:36:04.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:04.926+0000 7f7eccf82640 1 librados: shutdown 2024-03-25T19:36:04.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:11.755 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:11.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.869+0000 7f5c0728b640 10 monclient: get_monmap_and_config 2024-03-25T19:36:11.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.869+0000 7f5c0728b640 10 monclient: build_initial_monmap 2024-03-25T19:36:11.870 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.869+0000 7f5c0728b640 10 monclient: monmap: 2024-03-25T19:36:11.870 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:11.870 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:11.870 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:36:11.871109+0000 2024-03-25T19:36:11.870 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:36:11.871109+0000 2024-03-25T19:36:11.870 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:11.870 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:11.870 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:11.871 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:36:11.871 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:36:11.871 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 1 Processor -- start 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 1 -- start start 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 10 monclient: init 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c000b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 10 monclient(hunting): picked mon.noname-a con 0x7f5c000b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 10 monclient(hunting): picked mon.noname-c con 0x7f5c0014be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000af940 con 0x7f5c000b1730 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000b0a70 con 0x7f5c0014be90 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000b0d50 con 0x7f5c000b4d60 2024-03-25T19:36:11.872 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:11.873 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.870+0000 7f5c0728b640 10 monclient(hunting): authenticate will time out at 6420.051896s 2024-03-25T19:36:11.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.878+0000 7f5c05828640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f5c0014be90 0x7f5c0014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:59926/0 (socket says 172.21.15.52:59926) 2024-03-25T19:36:11.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.878+0000 7f5c05828640 1 -- 172.21.15.52:0/1015735889 learned_addr learned my addr 172.21.15.52:0/1015735889 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:11.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.901+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741790627 0 0) 0x7f5c000b0d50 con 0x7f5c000b4d60 2024-03-25T19:36:11.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.901+0000 7f5bf7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.901+0000 7f5bf7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.901+0000 7f5bf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.901+0000 7f5bf7fff640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:36:11.902 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.901+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bec003520 con 0x7f5c000b4d60 2024-03-25T19:36:11.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.902+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376346554 0 0) 0x7f5c000b0a70 con 0x7f5c0014be90 2024-03-25T19:36:11.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.902+0000 7f5bf7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.902+0000 7f5bf7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.902+0000 7f5bf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.902+0000 7f5bf7fff640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:36:11.903 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.902+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c000b0d50 con 0x7f5c0014be90 2024-03-25T19:36:11.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.906+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456482798 0 0) 0x7f5c000af940 con 0x7f5c000b1730 2024-03-25T19:36:11.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.906+0000 7f5bf7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.906+0000 7f5bf7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.906+0000 7f5bf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.906+0000 7f5bf7fff640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:36:11.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.906+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c000b0a70 con 0x7f5c000b1730 2024-03-25T19:36:11.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3100772373 0 0) 0x7f5c000b0d50 con 0x7f5c0014be90 2024-03-25T19:36:11.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 >> v1:172.21.15.52:6790/0 conn(0x7f5c000b4d60 legacy=0x7f5c0013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 >> v1:172.21.15.52:6789/0 conn(0x7f5c000b1730 legacy=0x7f5c000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:11.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 1 monclient: found mon.noname-c 2024-03-25T19:36:11.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c000b1030 con 0x7f5c0014be90 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 10 monclient: _finish_auth 0 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f5bf80019f0 con 0x7f5c0014be90 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5bf7fff640 10 monclient: dump: 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:11.938 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:11.939 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:11.939 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:11.939 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:11.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5c0728b640 5 monclient: authenticate success, global_id 4627 2024-03-25T19:36:11.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.936+0000 7f5c0728b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:11.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.939+0000 7f5bf7fff640 1 -- 172.21.15.52:0/1015735889 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bf8001b50 con 0x7f5c0014be90 2024-03-25T19:36:11.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.939+0000 7f5bf7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:11.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.939+0000 7f5c0728b640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:11.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.939+0000 7f5c0728b640 4 set_mon_vals no callback set 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.940+0000 7f5c0728b640 10 monclient: shutdown 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.940+0000 7f5c0728b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.940+0000 7f5c0728b640 1 -- 172.21.15.52:0/1015735889 >> v1:172.21.15.179:6789/0 conn(0x7f5c0014be90 legacy=0x7f5c0014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.940+0000 7f5c0728b640 1 -- 172.21.15.52:0/1015735889 shutdown_connections 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.940+0000 7f5c0728b640 1 -- 172.21.15.52:0/1015735889 >> 172.21.15.52:0/1015735889 conn(0x7f5c000617a0 msgr2=0x7f5c00061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.940+0000 7f5c0728b640 1 -- 172.21.15.52:0/1015735889 shutdown_connections 2024-03-25T19:36:11.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.940+0000 7f5c0728b640 1 -- 172.21.15.52:0/1015735889 wait complete. 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.941+0000 7f5c0728b640 10 monclient: build_initial_monmap 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.941+0000 7f5c0728b640 10 monclient: monmap: 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.941+0000 7f5c0728b640 1 librados: starting msgr at 2024-03-25T19:36:11.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.941+0000 7f5c0728b640 1 librados: starting objecter 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.941+0000 7f5c0728b640 1 Processor -- start 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.941+0000 7f5c0728b640 1 -- start start 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.941+0000 7f5c0728b640 1 librados: setting wanted keys 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.941+0000 7f5c0728b640 1 librados: calling monclient init 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.941+0000 7f5c0728b640 10 monclient: init 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.942+0000 7f5c0728b640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.942+0000 7f5c0728b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.942+0000 7f5c0728b640 10 monclient(hunting): picked mon.noname-a con 0x7f5c000b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.942+0000 7f5c0728b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c000b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.942+0000 7f5c0728b640 10 monclient(hunting): picked mon.noname-c con 0x7f5c0014be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.942+0000 7f5c0728b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c00141d70 con 0x7f5c000b1730 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.942+0000 7f5c0728b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c00143900 con 0x7f5c000b4d60 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.942+0000 7f5c0728b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c00142cb0 con 0x7f5c0014be90 2024-03-25T19:36:11.944 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.942+0000 7f5c0728b640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:11.945 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.942+0000 7f5c0728b640 10 monclient(hunting): authenticate will time out at 6420.123468s 2024-03-25T19:36:11.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.952+0000 7f5c04826640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f5c000b4d60 0x7f5c00132f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:59970/0 (socket says 172.21.15.52:59970) 2024-03-25T19:36:11.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.952+0000 7f5c04826640 1 -- 172.21.15.52:0/3570157539 learned_addr learned my addr 172.21.15.52:0/3570157539 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:11.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.955+0000 7f5bf5ffb640 10 client.?.objecter ms_handle_connect 0x7f5c000b4d60 2024-03-25T19:36:11.955 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.955+0000 7f5bf5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:11.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.962+0000 7f5bf5ffb640 10 client.?.objecter ms_handle_connect 0x7f5c000b1730 2024-03-25T19:36:11.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.962+0000 7f5bf5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:11.966 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.966+0000 7f5bf5ffb640 10 client.?.objecter ms_handle_connect 0x7f5c0014be90 2024-03-25T19:36:11.966 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.966+0000 7f5bf5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:11.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.972+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869105395 0 0) 0x7f5c00143900 con 0x7f5c000b4d60 2024-03-25T19:36:11.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.972+0000 7f5bf5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.972+0000 7f5bf5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.972+0000 7f5bf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.972+0000 7f5bf5ffb640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:36:11.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.972+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be4004eb0 con 0x7f5c000b4d60 2024-03-25T19:36:11.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.979+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076640612 0 0) 0x7f5c00141d70 con 0x7f5c000b1730 2024-03-25T19:36:11.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.979+0000 7f5bf5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.979+0000 7f5bf5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.979+0000 7f5bf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.979+0000 7f5bf5ffb640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:36:11.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.979+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c00143900 con 0x7f5c000b1730 2024-03-25T19:36:11.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.987+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844892170 0 0) 0x7f5c00142cb0 con 0x7f5c0014be90 2024-03-25T19:36:11.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.987+0000 7f5bf5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:36:11.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.987+0000 7f5bf5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:11.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.987+0000 7f5bf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:11.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.988+0000 7f5bf5ffb640 10 monclient(hunting): my global_id is 4613 2024-03-25T19:36:11.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.988+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c00141d70 con 0x7f5c0014be90 2024-03-25T19:36:11.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.991+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2212281640 0 0) 0x7f5be4004eb0 con 0x7f5c000b4d60 2024-03-25T19:36:11.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.991+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c00142cb0 con 0x7f5c000b4d60 2024-03-25T19:36:11.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.991+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f5bf0001a10 con 0x7f5c000b4d60 2024-03-25T19:36:11.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.991+0000 7f5bf5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.991+0000 7f5bf5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.991+0000 7f5bf5ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:11.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:11.991+0000 7f5bf5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.011+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4128743647 0 0) 0x7f5c00143900 con 0x7f5c000b1730 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.011+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5be4004eb0 con 0x7f5c000b1730 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.011+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f5bfc001950 con 0x7f5c000b1730 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.011+0000 7f5bf5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.011+0000 7f5bf5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.011+0000 7f5bf5ffb640 10 monclient(hunting): dump: 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:12.012 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:12.013 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.013 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:12.013 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:12.013 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:12.013 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:12.013 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1895838688 0 0) 0x7f5c00142cb0 con 0x7f5c000b4d60 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 >> v1:172.21.15.52:6790/0 conn(0x7f5c0014be90 legacy=0x7f5c00143d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 >> v1:172.21.15.52:6789/0 conn(0x7f5c000b1730 legacy=0x7f5c00141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5bf5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5bf5ffb640 1 monclient: found mon.b 2024-03-25T19:36:12.014 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5bf5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5bf5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c00140380 con 0x7f5c000b4d60 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5bf5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5bf5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5bf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5c0728b640 5 monclient: authenticate success, global_id 4630 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5c0728b640 10 monclient: _renew_subs 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5c0728b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5c0728b640 1 -- 172.21.15.52:0/3570157539 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c00133630 con 0x7f5c000b4d60 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5c0728b640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5c0728b640 10 monclient: _renew_subs 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5c0728b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5c0728b640 1 -- 172.21.15.52:0/3570157539 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c00140aa0 con 0x7f5c000b4d60 2024-03-25T19:36:12.015 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.013+0000 7f5c0728b640 1 librados: init done 2024-03-25T19:36:12.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.014+0000 7f5c0728b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.014+0000 7f5c0728b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.014+0000 7f5c0728b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:12.016 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.014+0000 7f5c0728b640 1 -- 172.21.15.52:0/3570157539 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c000627b0 con 0x7f5c000b4d60 2024-03-25T19:36:12.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.325+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bf0001c70 con 0x7f5c000b4d60 2024-03-25T19:36:12.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.325+0000 7f5bf5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:12.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.325+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f5bf0003a60 con 0x7f5c000b4d60 2024-03-25T19:36:12.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.325+0000 7f5bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:12.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.325+0000 7f5bf5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:12.326 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.325+0000 7f5bf5ffb640 10 monclient: dump: 2024-03-25T19:36:12.326 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:12.327 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:12.327 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:12.327 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:12.327 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:12.327 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:12.327 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:12.327 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:12.327 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:12.327 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:12.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.325+0000 7f5bf77fe640 4 set_mon_vals no callback set 2024-03-25T19:36:12.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.326+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f5bf0003c80 con 0x7f5c000b4d60 2024-03-25T19:36:12.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.326+0000 7f5c05027640 10 monclient: get_auth_request con 0x7f5be406df70 auth_method 0 2024-03-25T19:36:12.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.327+0000 7f5bf5ffb640 10 client.4630.objecter ms_handle_connect 0x7f5be406df70 2024-03-25T19:36:12.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.330+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f5bf0054820 con 0x7f5c000b4d60 2024-03-25T19:36:12.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.330+0000 7f5bf5ffb640 10 client.4630.objecter ms_dispatch 0x7f5c00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:12.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.330+0000 7f5bf5ffb640 3 client.4630.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:12.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.330+0000 7f5bf5ffb640 3 client.4630.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:12.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.330+0000 7f5bf5ffb640 20 client.4630.objecter dump_active .. 0 homeless 2024-03-25T19:36:12.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.330+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5bf0050330 con 0x7f5c000b4d60 2024-03-25T19:36:12.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.330+0000 7f5bf5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:12.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.330+0000 7f5bf5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.563+0000 7f5c0728b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.563+0000 7f5c0728b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.563+0000 7f5c0728b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:12.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.563+0000 7f5c0728b640 1 -- 172.21.15.52:0/3570157539 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c0013f710 con 0x7f5c000b4d60 2024-03-25T19:36:12.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.649+0000 7f5bf5ffb640 1 -- 172.21.15.52:0/3570157539 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f5bf0052260 con 0x7f5c000b4d60 2024-03-25T19:36:12.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.649+0000 7f5bf5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:12.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.649+0000 7f5bf5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:12.650 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:36:12.650 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:12.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 10 librados: watch_flush enter 2024-03-25T19:36:12.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 10 librados: watch_flush exit 2024-03-25T19:36:12.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 20 client.4630.objecter shutdown clearing up homeless session... 2024-03-25T19:36:12.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 10 client.4630.objecter successfully canceled tick 2024-03-25T19:36:12.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 1 -- 172.21.15.52:0/3570157539 >> v1:172.21.15.52:6800/3967427147 conn(0x7f5be406df70 legacy=0x7f5be4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 10 monclient: shutdown 2024-03-25T19:36:12.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:12.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 1 -- 172.21.15.52:0/3570157539 >> v1:172.21.15.179:6789/0 conn(0x7f5c000b4d60 legacy=0x7f5c00132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:12.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 1 -- 172.21.15.52:0/3570157539 shutdown_connections 2024-03-25T19:36:12.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 1 -- 172.21.15.52:0/3570157539 >> 172.21.15.52:0/3570157539 conn(0x7f5c000617a0 msgr2=0x7f5c000b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:12.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 1 -- 172.21.15.52:0/3570157539 shutdown_connections 2024-03-25T19:36:12.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 1 -- 172.21.15.52:0/3570157539 wait complete. 2024-03-25T19:36:12.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:12.651+0000 7f5c0728b640 1 librados: shutdown 2024-03-25T19:36:12.664 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:17.740 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:09:11.338 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.750 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:09:11.349 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.779 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:09:11.378 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.793 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:09:11.335 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.797 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:09:11.339 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.798 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:09:11.396 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.813 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:09:11.412 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.818 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:09:11.360 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:17.834 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:09:11.433 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:19.471 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:19.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.581+0000 7f28a9165640 10 monclient: get_monmap_and_config 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.581+0000 7f28a9165640 10 monclient: build_initial_monmap 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.581+0000 7f28a9165640 10 monclient: monmap: 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:36:19.582856+0000 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:36:19.582856+0000 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:36:19.582 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:19.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.582+0000 7f28a9165640 1 Processor -- start 2024-03-25T19:36:19.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.582+0000 7f28a9165640 1 -- start start 2024-03-25T19:36:19.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.582+0000 7f28a9165640 10 monclient: init 2024-03-25T19:36:19.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.583+0000 7f28a9165640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:19.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.583+0000 7f28a9165640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:19.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.583+0000 7f28a9165640 10 monclient(hunting): picked mon.noname-c con 0x7f28a40a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:19.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.583+0000 7f28a9165640 10 monclient(hunting): picked mon.noname-a con 0x7f28a413f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:19.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.583+0000 7f28a9165640 10 monclient(hunting): picked mon.noname-b con 0x7f28a41430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:19.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.583+0000 7f28a9165640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a4142520 con 0x7f28a413f710 2024-03-25T19:36:19.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.583+0000 7f28a9165640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a40b28a0 con 0x7f28a40a98f0 2024-03-25T19:36:19.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.583+0000 7f28a9165640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a40b2a30 con 0x7f28a41430c0 2024-03-25T19:36:19.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.583+0000 7f28a9165640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:19.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.583+0000 7f28a9165640 10 monclient(hunting): authenticate will time out at 6427.764155s 2024-03-25T19:36:19.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.595+0000 7f28a359e640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f28a41430c0 0x7f28a414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:36360/0 (socket says 172.21.15.52:36360) 2024-03-25T19:36:19.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.595+0000 7f28a359e640 1 -- 172.21.15.52:0/1551236614 learned_addr learned my addr 172.21.15.52:0/1551236614 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:19.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.604+0000 7f28a259c640 1 -- 172.21.15.52:0/1551236614 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757056602 0 0) 0x7f28a40b2a30 con 0x7f28a41430c0 2024-03-25T19:36:19.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.604+0000 7f28a259c640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.604+0000 7f28a259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.604+0000 7f28a259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.605 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.604+0000 7f28a259c640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:36:19.606 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.604+0000 7f28a259c640 1 -- 172.21.15.52:0/1551236614 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2890003520 con 0x7f28a41430c0 2024-03-25T19:36:19.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.615+0000 7f28a259c640 1 -- 172.21.15.52:0/1551236614 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724189032 0 0) 0x7f28a40b28a0 con 0x7f28a40a98f0 2024-03-25T19:36:19.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.615+0000 7f28a259c640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.615+0000 7f28a259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.615+0000 7f28a259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.615+0000 7f28a259c640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:36:19.615 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.615+0000 7f28a259c640 1 -- 172.21.15.52:0/1551236614 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a40b2a30 con 0x7f28a40a98f0 2024-03-25T19:36:19.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 1 -- 172.21.15.52:0/1551236614 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1218641697 0 0) 0x7f2890003520 con 0x7f28a41430c0 2024-03-25T19:36:19.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 1 -- 172.21.15.52:0/1551236614 >> v1:172.21.15.179:6789/0 conn(0x7f28a40a98f0 legacy=0x7f28a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 1 -- 172.21.15.52:0/1551236614 >> v1:172.21.15.52:6789/0 conn(0x7f28a413f710 legacy=0x7f28a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 10 monclient: _finish_hunting 0 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 1 monclient: found mon.noname-b 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 1 -- 172.21.15.52:0/1551236614 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a40b44c0 con 0x7f28a41430c0 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 10 monclient: _finish_auth 0 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 10 monclient: _check_auth_tickets 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 1 -- 172.21.15.52:0/1551236614 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f289c001a10 con 0x7f28a41430c0 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a259c640 10 monclient: dump: 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:19.625 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a9165640 5 monclient: authenticate success, global_id 4616 2024-03-25T19:36:19.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.624+0000 7f28a9165640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:19.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.637+0000 7f28a259c640 1 -- 172.21.15.52:0/1551236614 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f289c001b70 con 0x7f28a41430c0 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.637+0000 7f28a259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.637+0000 7f28a259c640 1 -- 172.21.15.52:0/1551236614 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f289c001f80 con 0x7f28a41430c0 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.637+0000 7f28a9165640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.637+0000 7f28a259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.637+0000 7f28a259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.637+0000 7f28a259c640 10 monclient: dump: 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:19.639 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:19.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.637+0000 7f28a9165640 4 set_mon_vals no callback set 2024-03-25T19:36:19.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.638+0000 7f28a9165640 10 monclient: shutdown 2024-03-25T19:36:19.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.638+0000 7f28a9165640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:19.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.638+0000 7f28a9165640 1 -- 172.21.15.52:0/1551236614 >> v1:172.21.15.52:6790/0 conn(0x7f28a41430c0 legacy=0x7f28a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.638+0000 7f28a9165640 1 -- 172.21.15.52:0/1551236614 shutdown_connections 2024-03-25T19:36:19.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.638+0000 7f28a9165640 1 -- 172.21.15.52:0/1551236614 >> 172.21.15.52:0/1551236614 conn(0x7f28a40b7ea0 msgr2=0x7f28a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.638+0000 7f28a9165640 1 -- 172.21.15.52:0/1551236614 shutdown_connections 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.639+0000 7f28a9165640 1 -- 172.21.15.52:0/1551236614 wait complete. 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.640+0000 7f28a9165640 10 monclient: build_initial_monmap 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.640+0000 7f28a9165640 10 monclient: monmap: 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:36:19.641 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:19.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.641+0000 7f28a9165640 1 librados: starting msgr at 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.641+0000 7f28a9165640 1 librados: starting objecter 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.641+0000 7f28a9165640 1 Processor -- start 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.641+0000 7f28a9165640 1 -- start start 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.641+0000 7f28a9165640 1 librados: setting wanted keys 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.641+0000 7f28a9165640 1 librados: calling monclient init 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.641+0000 7f28a9165640 10 monclient: init 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.642+0000 7f28a9165640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.642+0000 7f28a9165640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.642+0000 7f28a9165640 10 monclient(hunting): picked mon.noname-a con 0x7f28a40a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.642+0000 7f28a9165640 10 monclient(hunting): picked mon.noname-b con 0x7f28a413f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.642+0000 7f28a9165640 10 monclient(hunting): picked mon.noname-c con 0x7f28a41430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.642+0000 7f28a9165640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a40704d0 con 0x7f28a40a98f0 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.642+0000 7f28a9165640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a41df0d0 con 0x7f28a413f710 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.642+0000 7f28a9165640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a406f490 con 0x7f28a41430c0 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.642+0000 7f28a9165640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:19.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.642+0000 7f28a9165640 10 monclient(hunting): authenticate will time out at 6427.823125s 2024-03-25T19:36:19.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.652+0000 7f289bfff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f28a413f710 0x7f28a41df5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:56682/0 (socket says 172.21.15.52:56682) 2024-03-25T19:36:19.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.652+0000 7f289bfff640 1 -- 172.21.15.52:0/184071841 learned_addr learned my addr 172.21.15.52:0/184071841 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:19.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.654+0000 7f289b7fe640 10 client.?.objecter ms_handle_connect 0x7f28a413f710 2024-03-25T19:36:19.654 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.654+0000 7f289b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:19.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.659+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418544506 0 0) 0x7f28a41df0d0 con 0x7f28a413f710 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.659+0000 7f289b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.659+0000 7f289b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.659+0000 7f289b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.659+0000 7f289b7fe640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.659+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2884004eb0 con 0x7f28a413f710 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.660+0000 7f289b7fe640 10 client.?.objecter ms_handle_connect 0x7f28a41430c0 2024-03-25T19:36:19.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.660+0000 7f289b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:19.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.662+0000 7f289b7fe640 10 client.?.objecter ms_handle_connect 0x7f28a40a98f0 2024-03-25T19:36:19.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.662+0000 7f289b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:19.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.667+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213381177 0 0) 0x7f28a406f490 con 0x7f28a41430c0 2024-03-25T19:36:19.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.667+0000 7f289b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.667+0000 7f289b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.667+0000 7f289b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.667+0000 7f289b7fe640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:36:19.667 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.667+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a41df0d0 con 0x7f28a41430c0 2024-03-25T19:36:19.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.668+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957548640 0 0) 0x7f28a40704d0 con 0x7f28a40a98f0 2024-03-25T19:36:19.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.668+0000 7f289b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:19.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.668+0000 7f289b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:19.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.668+0000 7f289b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:19.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.668+0000 7f289b7fe640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:36:19.669 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.668+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a406f490 con 0x7f28a40a98f0 2024-03-25T19:36:19.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.678+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1412020649 0 0) 0x7f2884004eb0 con 0x7f28a413f710 2024-03-25T19:36:19.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.678+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28a40704d0 con 0x7f28a413f710 2024-03-25T19:36:19.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.678+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f288c0019d0 con 0x7f28a413f710 2024-03-25T19:36:19.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.678+0000 7f289b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.678+0000 7f289b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:19.679 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.678+0000 7f289b7fe640 10 monclient(hunting): dump: 2024-03-25T19:36:19.679 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:19.679 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:19.679 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:19.679 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:19.680 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.680 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:19.680 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:19.680 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:19.680 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:19.680 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:19.680 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.678+0000 7f289b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:19.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.686+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 306274129 0 0) 0x7f28a40704d0 con 0x7f28a413f710 2024-03-25T19:36:19.687 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.686+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 >> v1:172.21.15.52:6790/0 conn(0x7f28a41430c0 legacy=0x7f28a4138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.686+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 >> v1:172.21.15.52:6789/0 conn(0x7f28a40a98f0 legacy=0x7f28a406ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.686+0000 7f289b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.686+0000 7f289b7fe640 1 monclient: found mon.b 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.686+0000 7f289b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f289b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a41dfcc0 con 0x7f28a413f710 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f289b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f289b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f289b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f28a9165640 5 monclient: authenticate success, global_id 4636 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f28a9165640 10 monclient: _renew_subs 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f28a9165640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f28a9165640 1 -- 172.21.15.52:0/184071841 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a406e0e0 con 0x7f28a413f710 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f28a9165640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f28a9165640 10 monclient: _renew_subs 2024-03-25T19:36:19.688 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f28a9165640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:19.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f28a9165640 1 -- 172.21.15.52:0/184071841 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a4070db0 con 0x7f28a413f710 2024-03-25T19:36:19.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f28a9165640 1 librados: init done 2024-03-25T19:36:19.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.687+0000 7f28a9165640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:19.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.688+0000 7f28a9165640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:19.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.688+0000 7f28a9165640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:19.689 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.688+0000 7f28a9165640 1 -- 172.21.15.52:0/184071841 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a41def30 con 0x7f28a413f710 2024-03-25T19:36:19.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.969+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f288c0035f0 con 0x7f28a413f710 2024-03-25T19:36:19.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.969+0000 7f289b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:19.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.969+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f288c003860 con 0x7f28a413f710 2024-03-25T19:36:19.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.969+0000 7f289b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:19.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.969+0000 7f289b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:19.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.969+0000 7f289b7fe640 10 monclient: dump: 2024-03-25T19:36:19.970 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:19.970 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:19.970 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:19.970 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:19.970 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:19.971 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:19.971 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:19.971 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:19.971 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:19.971 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:19.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.969+0000 7f28a1d9b640 4 set_mon_vals no callback set 2024-03-25T19:36:19.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.970+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f288c003a40 con 0x7f28a413f710 2024-03-25T19:36:19.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.970+0000 7f28a2d9d640 10 monclient: get_auth_request con 0x7f288406dda0 auth_method 0 2024-03-25T19:36:19.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.971+0000 7f289b7fe640 10 client.4636.objecter ms_handle_connect 0x7f288406dda0 2024-03-25T19:36:19.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.975+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f288c0546c0 con 0x7f28a413f710 2024-03-25T19:36:19.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.975+0000 7f289b7fe640 10 client.4636.objecter ms_dispatch 0x7f28a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:19.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.975+0000 7f289b7fe640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:19.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.975+0000 7f289b7fe640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:19.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.975+0000 7f289b7fe640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T19:36:19.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.975+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f288c0501d0 con 0x7f28a413f710 2024-03-25T19:36:19.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.975+0000 7f289b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:19.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:19.975+0000 7f289b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:20.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.211+0000 7f28a9165640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.211+0000 7f28a9165640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.211+0000 7f28a9165640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:20.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.211+0000 7f28a9165640 1 -- 172.21.15.52:0/184071841 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28a41dfe50 con 0x7f28a413f710 2024-03-25T19:36:20.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.246+0000 7f289b7fe640 1 -- 172.21.15.52:0/184071841 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f288c052100 con 0x7f28a413f710 2024-03-25T19:36:20.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.246+0000 7f289b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:20.247 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.246+0000 7f289b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:20.247 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:36:20.247 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:20.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 10 librados: watch_flush enter 2024-03-25T19:36:20.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 10 librados: watch_flush exit 2024-03-25T19:36:20.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T19:36:20.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 10 client.4636.objecter successfully canceled tick 2024-03-25T19:36:20.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 1 -- 172.21.15.52:0/184071841 >> v1:172.21.15.52:6800/3967427147 conn(0x7f288406dda0 legacy=0x7f2884070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 10 monclient: shutdown 2024-03-25T19:36:20.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:20.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 1 -- 172.21.15.52:0/184071841 >> v1:172.21.15.179:6789/0 conn(0x7f28a413f710 legacy=0x7f28a41df5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:20.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 1 -- 172.21.15.52:0/184071841 shutdown_connections 2024-03-25T19:36:20.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 1 -- 172.21.15.52:0/184071841 >> 172.21.15.52:0/184071841 conn(0x7f28a40b7ea0 msgr2=0x7f28a414e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:20.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 1 -- 172.21.15.52:0/184071841 shutdown_connections 2024-03-25T19:36:20.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 1 -- 172.21.15.52:0/184071841 wait complete. 2024-03-25T19:36:20.250 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:20.248+0000 7f28a9165640 1 librados: shutdown 2024-03-25T19:36:20.261 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:27.068 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:27.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.186+0000 7fc0547f6640 10 monclient: get_monmap_and_config 2024-03-25T19:36:27.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.186+0000 7fc0547f6640 10 monclient: build_initial_monmap 2024-03-25T19:36:27.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.186+0000 7fc0547f6640 10 monclient: monmap: 2024-03-25T19:36:27.186 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:27.186 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:27.186 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:36:27.187059+0000 2024-03-25T19:36:27.186 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:36:27.187059+0000 2024-03-25T19:36:27.187 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:27.187 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:27.187 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:27.187 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:36:27.187 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:36:27.187 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.187+0000 7fc0547f6640 1 Processor -- start 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.187+0000 7fc0547f6640 1 -- start start 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.187+0000 7fc0547f6640 10 monclient: init 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.187+0000 7fc0547f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.187+0000 7fc0547f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.187+0000 7fc0547f6640 10 monclient(hunting): picked mon.noname-a con 0x7fc04c0b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.188+0000 7fc0547f6640 10 monclient(hunting): picked mon.noname-b con 0x7fc04c0b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.188+0000 7fc0547f6640 10 monclient(hunting): picked mon.noname-c con 0x7fc04c14be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.188+0000 7fc0547f6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c0af940 con 0x7fc04c0b4d60 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.188+0000 7fc0547f6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c0b0a70 con 0x7fc04c14be90 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.188+0000 7fc0547f6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c0b0d50 con 0x7fc04c0b1730 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.188+0000 7fc0547f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:27.188 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.188+0000 7fc0547f6640 10 monclient(hunting): authenticate will time out at 6435.368164s 2024-03-25T19:36:27.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.192+0000 7fc052d93640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc04c14be90 0x7fc04c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:58066/0 (socket says 172.21.15.52:58066) 2024-03-25T19:36:27.191 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.192+0000 7fc052d93640 1 -- 172.21.15.52:0/1806606581 learned_addr learned my addr 172.21.15.52:0/1806606581 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:27.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.203+0000 7fc051590640 1 -- 172.21.15.52:0/1806606581 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651665753 0 0) 0x7fc04c0b0a70 con 0x7fc04c14be90 2024-03-25T19:36:27.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.203+0000 7fc051590640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.203+0000 7fc051590640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.203+0000 7fc051590640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.202 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.203+0000 7fc051590640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:36:27.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.203+0000 7fc051590640 1 -- 172.21.15.52:0/1806606581 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc038003520 con 0x7fc04c14be90 2024-03-25T19:36:27.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 1 -- 172.21.15.52:0/1806606581 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2538051483 0 0) 0x7fc038003520 con 0x7fc04c14be90 2024-03-25T19:36:27.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 1 -- 172.21.15.52:0/1806606581 >> v1:172.21.15.52:6790/0 conn(0x7fc04c0b1730 legacy=0x7fc04c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 1 -- 172.21.15.52:0/1806606581 >> v1:172.21.15.52:6789/0 conn(0x7fc04c0b4d60 legacy=0x7fc04c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 10 monclient: _finish_hunting 0 2024-03-25T19:36:27.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 1 monclient: found mon.noname-c 2024-03-25T19:36:27.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:27.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:36:27.212 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 1 -- 172.21.15.52:0/1806606581 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c0b1030 con 0x7fc04c14be90 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 10 monclient: _finish_auth 0 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 10 monclient: _check_auth_tickets 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 1 -- 172.21.15.52:0/1806606581 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc044001a10 con 0x7fc04c14be90 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc051590640 10 monclient: dump: 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:27.213 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:27.214 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:27.214 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:27.214 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:27.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc0547f6640 5 monclient: authenticate success, global_id 4639 2024-03-25T19:36:27.214 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.212+0000 7fc0547f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:27.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.233+0000 7fc051590640 1 -- 172.21.15.52:0/1806606581 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc044001b70 con 0x7fc04c14be90 2024-03-25T19:36:27.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.233+0000 7fc051590640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.233+0000 7fc051590640 1 -- 172.21.15.52:0/1806606581 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc044001f80 con 0x7fc04c14be90 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.233+0000 7fc051590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.233+0000 7fc051590640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.233+0000 7fc051590640 10 monclient: dump: 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:27.233 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:27.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.233+0000 7fc0547f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:27.234 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.233+0000 7fc0547f6640 4 set_mon_vals no callback set 2024-03-25T19:36:27.235 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.234+0000 7fc0547f6640 10 monclient: shutdown 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.234+0000 7fc0547f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.234+0000 7fc0547f6640 1 -- 172.21.15.52:0/1806606581 >> v1:172.21.15.179:6789/0 conn(0x7fc04c14be90 legacy=0x7fc04c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.234+0000 7fc0547f6640 1 -- 172.21.15.52:0/1806606581 shutdown_connections 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.234+0000 7fc0547f6640 1 -- 172.21.15.52:0/1806606581 >> 172.21.15.52:0/1806606581 conn(0x7fc04c0617a0 msgr2=0x7fc04c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.234+0000 7fc0547f6640 1 -- 172.21.15.52:0/1806606581 shutdown_connections 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.234+0000 7fc0547f6640 1 -- 172.21.15.52:0/1806606581 wait complete. 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.235+0000 7fc0547f6640 10 monclient: build_initial_monmap 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.235+0000 7fc0547f6640 10 monclient: monmap: 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:36:27.236 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.235+0000 7fc0547f6640 1 librados: starting msgr at 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.235+0000 7fc0547f6640 1 librados: starting objecter 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.235+0000 7fc0547f6640 1 Processor -- start 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.235+0000 7fc0547f6640 1 -- start start 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.235+0000 7fc0547f6640 1 librados: setting wanted keys 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.235+0000 7fc0547f6640 1 librados: calling monclient init 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.235+0000 7fc0547f6640 10 monclient: init 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.236+0000 7fc0547f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.236+0000 7fc0547f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.236+0000 7fc0547f6640 10 monclient(hunting): picked mon.noname-a con 0x7fc04c0b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.236+0000 7fc0547f6640 10 monclient(hunting): picked mon.noname-b con 0x7fc04c0b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.236+0000 7fc0547f6640 10 monclient(hunting): picked mon.noname-c con 0x7fc04c14be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:27.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.236+0000 7fc0547f6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c140c10 con 0x7fc04c0b1730 2024-03-25T19:36:27.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.236+0000 7fc0547f6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c134900 con 0x7fc04c0b4d60 2024-03-25T19:36:27.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.236+0000 7fc0547f6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c141a70 con 0x7fc04c14be90 2024-03-25T19:36:27.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.236+0000 7fc0547f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:27.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.236+0000 7fc0547f6640 10 monclient(hunting): authenticate will time out at 6435.416246s 2024-03-25T19:36:27.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.247+0000 7fc051d91640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc04c0b4d60 0x7fc04c133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:58106/0 (socket says 172.21.15.52:58106) 2024-03-25T19:36:27.246 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.247+0000 7fc051d91640 1 -- 172.21.15.52:0/295703957 learned_addr learned my addr 172.21.15.52:0/295703957 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:27.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.250+0000 7fc042ffd640 10 client.?.objecter ms_handle_connect 0x7fc04c0b4d60 2024-03-25T19:36:27.249 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.250+0000 7fc042ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.252+0000 7fc042ffd640 10 client.?.objecter ms_handle_connect 0x7fc04c14be90 2024-03-25T19:36:27.252 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.253+0000 7fc042ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.257+0000 7fc042ffd640 10 client.?.objecter ms_handle_connect 0x7fc04c0b1730 2024-03-25T19:36:27.256 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.257+0000 7fc042ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.261+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255670060 0 0) 0x7fc04c141a70 con 0x7fc04c14be90 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.261+0000 7fc042ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.261+0000 7fc042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.261+0000 7fc042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.261+0000 7fc042ffd640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:36:27.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.262+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc030004eb0 con 0x7fc04c14be90 2024-03-25T19:36:27.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.265+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110491017 0 0) 0x7fc04c140c10 con 0x7fc04c0b1730 2024-03-25T19:36:27.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.265+0000 7fc042ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.265+0000 7fc042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.265+0000 7fc042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.265+0000 7fc042ffd640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:36:27.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.265+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc04c141a70 con 0x7fc04c0b1730 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.266+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689637532 0 0) 0x7fc04c134900 con 0x7fc04c0b4d60 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.266+0000 7fc042ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.266+0000 7fc042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.266+0000 7fc042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.266+0000 7fc042ffd640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:36:27.266 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.266+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc04c140c10 con 0x7fc04c0b4d60 2024-03-25T19:36:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.270+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2739292363 0 0) 0x7fc030004eb0 con 0x7fc04c14be90 2024-03-25T19:36:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.270+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc04c134900 con 0x7fc04c14be90 2024-03-25T19:36:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.270+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc044001e80 con 0x7fc04c14be90 2024-03-25T19:36:27.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.270+0000 7fc042ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.271+0000 7fc042ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.271+0000 7fc042ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:27.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.271+0000 7fc042ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:27.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.278+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4234006724 0 0) 0x7fc04c141a70 con 0x7fc04c0b1730 2024-03-25T19:36:27.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.278+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc030004eb0 con 0x7fc04c0b1730 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.278+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc0480018b0 con 0x7fc04c0b1730 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.278+0000 7fc042ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.278+0000 7fc042ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.278+0000 7fc042ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:27.278 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:27.279 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:27.279 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4086383426 0 0) 0x7fc04c134900 con 0x7fc04c14be90 2024-03-25T19:36:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 >> v1:172.21.15.179:6789/0 conn(0x7fc04c0b4d60 legacy=0x7fc04c133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 >> v1:172.21.15.52:6789/0 conn(0x7fc04c0b1730 legacy=0x7fc04c142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc042ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc042ffd640 1 monclient: found mon.c 2024-03-25T19:36:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc042ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:36:27.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c141d50 con 0x7fc04c14be90 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc042ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc042ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc0547f6640 5 monclient: authenticate success, global_id 4622 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc0547f6640 10 monclient: _renew_subs 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc0547f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc0547f6640 1 -- 172.21.15.52:0/295703957 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04c140380 con 0x7fc04c14be90 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc0547f6640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc0547f6640 10 monclient: _renew_subs 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc0547f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc0547f6640 1 -- 172.21.15.52:0/295703957 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04c1389e0 con 0x7fc04c14be90 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.279+0000 7fc0547f6640 1 librados: init done 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.280+0000 7fc0547f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:27.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.280+0000 7fc0547f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:27.282 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.280+0000 7fc0547f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:36:27.282 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.280+0000 7fc0547f6640 1 -- 172.21.15.52:0/295703957 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04c0627b0 con 0x7fc04c14be90 2024-03-25T19:36:27.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.610+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc044002020 con 0x7fc04c14be90 2024-03-25T19:36:27.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.610+0000 7fc042ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:27.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.610+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc044004fb0 con 0x7fc04c14be90 2024-03-25T19:36:27.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.610+0000 7fc042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:27.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.610+0000 7fc042ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:27.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.610+0000 7fc042ffd640 10 monclient: dump: 2024-03-25T19:36:27.610 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:27.610 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:27.610 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:27.610 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:27.611 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:27.611 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:27.611 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:27.611 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:27.611 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:27.611 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:27.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.610+0000 7fc050d8f640 4 set_mon_vals no callback set 2024-03-25T19:36:27.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.611+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fc04401af80 con 0x7fc04c14be90 2024-03-25T19:36:27.611 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.611+0000 7fc052592640 10 monclient: get_auth_request con 0x7fc03006df20 auth_method 0 2024-03-25T19:36:27.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.612+0000 7fc042ffd640 10 client.4622.objecter ms_handle_connect 0x7fc03006df20 2024-03-25T19:36:27.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.618+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fc044054a80 con 0x7fc04c14be90 2024-03-25T19:36:27.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.618+0000 7fc042ffd640 10 client.4622.objecter ms_dispatch 0x7fc04c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:27.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.618+0000 7fc042ffd640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:27.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.618+0000 7fc042ffd640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:27.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.618+0000 7fc042ffd640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-25T19:36:27.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.618+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0440503c0 con 0x7fc04c14be90 2024-03-25T19:36:27.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.618+0000 7fc042ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:27.618 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.618+0000 7fc042ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:27.865 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.866+0000 7fc0547f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:27.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.866+0000 7fc0547f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:27.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.866+0000 7fc0547f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:36:27.866 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.866+0000 7fc0547f6640 1 -- 172.21.15.52:0/295703957 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc04c140930 con 0x7fc04c14be90 2024-03-25T19:36:27.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.888+0000 7fc042ffd640 1 -- 172.21.15.52:0/295703957 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fc04c140930 con 0x7fc04c14be90 2024-03-25T19:36:27.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.889+0000 7fc042ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:27.888 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.889+0000 7fc042ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:27.888 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:36:27.889 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:27.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 10 librados: watch_flush enter 2024-03-25T19:36:27.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 10 librados: watch_flush exit 2024-03-25T19:36:27.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-25T19:36:27.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 10 client.4622.objecter successfully canceled tick 2024-03-25T19:36:27.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 1 -- 172.21.15.52:0/295703957 >> v1:172.21.15.52:6800/3967427147 conn(0x7fc03006df20 legacy=0x7fc0300703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.891 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 10 monclient: shutdown 2024-03-25T19:36:27.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:27.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 1 -- 172.21.15.52:0/295703957 >> v1:172.21.15.52:6790/0 conn(0x7fc04c14be90 legacy=0x7fc04c1341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:27.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 1 -- 172.21.15.52:0/295703957 shutdown_connections 2024-03-25T19:36:27.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 1 -- 172.21.15.52:0/295703957 >> 172.21.15.52:0/295703957 conn(0x7fc04c0617a0 msgr2=0x7fc04c13b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:27.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 1 -- 172.21.15.52:0/295703957 shutdown_connections 2024-03-25T19:36:27.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 1 -- 172.21.15.52:0/295703957 wait complete. 2024-03-25T19:36:27.892 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:27.891+0000 7fc0547f6640 1 librados: shutdown 2024-03-25T19:36:27.903 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:34.710 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:34.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.824+0000 7fa3f4f9d640 10 monclient: get_monmap_and_config 2024-03-25T19:36:34.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.824+0000 7fa3f4f9d640 10 monclient: build_initial_monmap 2024-03-25T19:36:34.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.824+0000 7fa3f4f9d640 10 monclient: monmap: 2024-03-25T19:36:34.823 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:34.824 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:34.824 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:36:34.824606+0000 2024-03-25T19:36:34.824 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:36:34.824606+0000 2024-03-25T19:36:34.824 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:34.824 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:34.824 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:34.824 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:36:34.824 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:36:34.824 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:34.824 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.824+0000 7fa3f4f9d640 1 Processor -- start 2024-03-25T19:36:34.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.825+0000 7fa3f4f9d640 1 -- start start 2024-03-25T19:36:34.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.825+0000 7fa3f4f9d640 10 monclient: init 2024-03-25T19:36:34.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.825+0000 7fa3f4f9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:34.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.825+0000 7fa3f4f9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:34.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.825+0000 7fa3f4f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa3f00a96e0 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:34.825 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.825+0000 7fa3f4f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fa3f00b1b10 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:34.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.825+0000 7fa3f4f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fa3f0146060 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:34.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.826+0000 7fa3f4f9d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f0140040 con 0x7fa3f0146060 2024-03-25T19:36:34.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.826+0000 7fa3f4f9d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f00b1fa0 con 0x7fa3f00a96e0 2024-03-25T19:36:34.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.826+0000 7fa3f4f9d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f00b24c0 con 0x7fa3f00b1b10 2024-03-25T19:36:34.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.826+0000 7fa3f4f9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:34.826 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.826+0000 7fa3f4f9d640 10 monclient(hunting): authenticate will time out at 6443.005952s 2024-03-25T19:36:34.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.838+0000 7fa3ee59c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fa3f00a96e0 0x7fa3f00b0ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:43390/0 (socket says 172.21.15.52:43390) 2024-03-25T19:36:34.837 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.838+0000 7fa3ee59c640 1 -- 172.21.15.52:0/527266784 learned_addr learned my addr 172.21.15.52:0/527266784 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:34.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.851+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873151694 0 0) 0x7fa3f00b1fa0 con 0x7fa3f00a96e0 2024-03-25T19:36:34.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.851+0000 7fa3edd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.851+0000 7fa3edd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.851+0000 7fa3edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.851+0000 7fa3edd9b640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:36:34.854 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.851+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3dc003520 con 0x7fa3f00a96e0 2024-03-25T19:36:34.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.874+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577023014 0 0) 0x7fa3f00b24c0 con 0x7fa3f00b1b10 2024-03-25T19:36:34.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.874+0000 7fa3edd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.874 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.874+0000 7fa3edd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.874+0000 7fa3edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.874+0000 7fa3edd9b640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:36:34.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.874+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f00b1fa0 con 0x7fa3f00b1b10 2024-03-25T19:36:34.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.875+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058001841 0 0) 0x7fa3f0140040 con 0x7fa3f0146060 2024-03-25T19:36:34.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.875+0000 7fa3edd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.875+0000 7fa3edd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.875+0000 7fa3edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.875+0000 7fa3edd9b640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:36:34.875 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.875+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f00b24c0 con 0x7fa3f0146060 2024-03-25T19:36:34.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3011309560 0 0) 0x7fa3dc003520 con 0x7fa3f00a96e0 2024-03-25T19:36:34.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 >> v1:172.21.15.52:6790/0 conn(0x7fa3f00b1b10 legacy=0x7fa3f0145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 >> v1:172.21.15.52:6789/0 conn(0x7fa3f0146060 legacy=0x7fa3f014dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:36:34.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 1 monclient: found mon.noname-c 2024-03-25T19:36:34.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:34.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:36:34.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f00b15b0 con 0x7fa3f00a96e0 2024-03-25T19:36:34.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 10 monclient: _finish_auth 0 2024-03-25T19:36:34.878 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa3e0001690 con 0x7fa3f00a96e0 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3edd9b640 10 monclient: dump: 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:34.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3f4f9d640 5 monclient: authenticate success, global_id 4645 2024-03-25T19:36:34.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.878+0000 7fa3f4f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:34.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.905+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3e00017f0 con 0x7fa3f00a96e0 2024-03-25T19:36:34.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.905+0000 7fa3edd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:34.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.905+0000 7fa3edd9b640 1 -- 172.21.15.52:0/527266784 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa3e0001c00 con 0x7fa3f00a96e0 2024-03-25T19:36:34.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.905+0000 7fa3f4f9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:34.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.905+0000 7fa3f4f9d640 4 set_mon_vals no callback set 2024-03-25T19:36:34.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.905+0000 7fa3edd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:34.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.905+0000 7fa3edd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:34.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.905+0000 7fa3edd9b640 10 monclient: dump: 2024-03-25T19:36:34.905 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:34.906 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:34.906 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:34.906 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:34.906 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:34.906 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:34.906 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:34.906 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:34.906 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:34.906 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:34.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.906+0000 7fa3f4f9d640 10 monclient: shutdown 2024-03-25T19:36:34.907 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.906+0000 7fa3f4f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.906+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/527266784 >> v1:172.21.15.179:6789/0 conn(0x7fa3f00a96e0 legacy=0x7fa3f00b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.906+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/527266784 shutdown_connections 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.906+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/527266784 >> 172.21.15.52:0/527266784 conn(0x7fa3f00b7ea0 msgr2=0x7fa3f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.906+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/527266784 shutdown_connections 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.907+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/527266784 wait complete. 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.907+0000 7fa3f4f9d640 10 monclient: build_initial_monmap 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.907+0000 7fa3f4f9d640 10 monclient: monmap: 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:36:34.908 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:36:34.909 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:34.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 1 librados: starting msgr at 2024-03-25T19:36:34.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 1 librados: starting objecter 2024-03-25T19:36:34.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 1 Processor -- start 2024-03-25T19:36:34.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 1 -- start start 2024-03-25T19:36:34.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 1 librados: setting wanted keys 2024-03-25T19:36:34.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 1 librados: calling monclient init 2024-03-25T19:36:34.909 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 10 monclient: init 2024-03-25T19:36:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa3f00a96e0 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fa3f0146060 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fa3f01db740 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f00aec60 con 0x7fa3f01db740 2024-03-25T19:36:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f00ac870 con 0x7fa3f0146060 2024-03-25T19:36:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f01dab10 con 0x7fa3f00a96e0 2024-03-25T19:36:34.910 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:34.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.908+0000 7fa3f4f9d640 10 monclient(hunting): authenticate will time out at 6443.088879s 2024-03-25T19:36:34.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.929+0000 7fa3ee59c640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fa3f00a96e0 0x7fa3f00ad5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:33246/0 (socket says 172.21.15.52:33246) 2024-03-25T19:36:34.928 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.929+0000 7fa3ee59c640 1 -- 172.21.15.52:0/3919897408 learned_addr learned my addr 172.21.15.52:0/3919897408 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:34.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.933+0000 7fa3e6ffd640 10 client.?.objecter ms_handle_connect 0x7fa3f00a96e0 2024-03-25T19:36:34.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.933+0000 7fa3e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:34.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.933+0000 7fa3e6ffd640 10 client.?.objecter ms_handle_connect 0x7fa3f0146060 2024-03-25T19:36:34.932 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.933+0000 7fa3e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:34.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.941+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606543763 0 0) 0x7fa3f00ac870 con 0x7fa3f0146060 2024-03-25T19:36:34.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.941+0000 7fa3e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.941+0000 7fa3e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.941+0000 7fa3e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.941+0000 7fa3e6ffd640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:36:34.941 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.941+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d0004eb0 con 0x7fa3f0146060 2024-03-25T19:36:34.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.942+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498105259 0 0) 0x7fa3f01dab10 con 0x7fa3f00a96e0 2024-03-25T19:36:34.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.943+0000 7fa3e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.943+0000 7fa3e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.943+0000 7fa3e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.943+0000 7fa3e6ffd640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:36:34.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.943+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f00ac870 con 0x7fa3f00a96e0 2024-03-25T19:36:34.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.943+0000 7fa3e6ffd640 10 client.?.objecter ms_handle_connect 0x7fa3f01db740 2024-03-25T19:36:34.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.943+0000 7fa3e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:34.956 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.957+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161725191 0 0) 0x7fa3f00aec60 con 0x7fa3f01db740 2024-03-25T19:36:34.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.957+0000 7fa3e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:34.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.957+0000 7fa3e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:34.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.957+0000 7fa3e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:34.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.957+0000 7fa3e6ffd640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:36:34.957 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.957+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f01dab10 con 0x7fa3f01db740 2024-03-25T19:36:34.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.960+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1978009548 0 0) 0x7fa3d0004eb0 con 0x7fa3f0146060 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.960+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3f00aec60 con 0x7fa3f0146060 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.960+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa3d80019e0 con 0x7fa3f0146060 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.960+0000 7fa3e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.960+0000 7fa3e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.960+0000 7fa3e6ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:34.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.960+0000 7fa3e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.965+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3202499704 0 0) 0x7fa3f00ac870 con 0x7fa3f00a96e0 2024-03-25T19:36:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.965+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3d0004eb0 con 0x7fa3f00a96e0 2024-03-25T19:36:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.965+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa3e0001b80 con 0x7fa3f00a96e0 2024-03-25T19:36:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.965+0000 7fa3e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.965+0000 7fa3e6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:34.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.965+0000 7fa3e6ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:34.965 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:34.965 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:34.966 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:34.966 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:34.966 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:34.966 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:34.966 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:34.966 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:34.966 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:34.966 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:34.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.970+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2015977198 0 0) 0x7fa3f01dab10 con 0x7fa3f01db740 2024-03-25T19:36:34.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.970+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3f00ac870 con 0x7fa3f01db740 2024-03-25T19:36:34.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.970+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa3e8001bd0 con 0x7fa3f01db740 2024-03-25T19:36:34.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.970+0000 7fa3e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:34.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.970+0000 7fa3e6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:34.970 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.970+0000 7fa3e6ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:34.970 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:34.970 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:34.970 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:34.971 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:34.971 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:34.971 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:34.971 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:34.971 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:34.971 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:34.971 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2026020646 0 0) 0x7fa3f00aec60 con 0x7fa3f0146060 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 >> v1:172.21.15.52:6790/0 conn(0x7fa3f00a96e0 legacy=0x7fa3f00ad5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 >> v1:172.21.15.52:6789/0 conn(0x7fa3f01db740 legacy=0x7fa3f013a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3e6ffd640 1 monclient: found mon.b 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3e6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f00add00 con 0x7fa3f0146060 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3e6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3f4f9d640 5 monclient: authenticate success, global_id 4648 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3f4f9d640 10 monclient: _renew_subs 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3f4f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:34.981 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/3919897408 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f013b180 con 0x7fa3f0146060 2024-03-25T19:36:34.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3f4f9d640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:34.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3f4f9d640 10 monclient: _renew_subs 2024-03-25T19:36:34.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3f4f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:34.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/3919897408 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f013ac60 con 0x7fa3f0146060 2024-03-25T19:36:34.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.980+0000 7fa3f4f9d640 1 librados: init done 2024-03-25T19:36:34.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.982+0000 7fa3f4f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:34.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.982+0000 7fa3f4f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:34.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.982+0000 7fa3f4f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:34.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:34.982+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/3919897408 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f01daca0 con 0x7fa3f0146060 2024-03-25T19:36:35.237 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.238+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3d80035f0 con 0x7fa3f0146060 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.238+0000 7fa3e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.238+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fa3d8003900 con 0x7fa3f0146060 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.238+0000 7fa3e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.238+0000 7fa3e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.238+0000 7fa3e6ffd640 10 monclient: dump: 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:35.238 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:35.239 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:35.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.238+0000 7fa3ed59a640 4 set_mon_vals no callback set 2024-03-25T19:36:35.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.239+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fa3d8019990 con 0x7fa3f0146060 2024-03-25T19:36:35.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.239+0000 7fa3ee59c640 10 monclient: get_auth_request con 0x7fa3d006e010 auth_method 0 2024-03-25T19:36:35.239 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.239+0000 7fa3e6ffd640 10 client.4648.objecter ms_handle_connect 0x7fa3d006e010 2024-03-25T19:36:35.243 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.244+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fa3d80535f0 con 0x7fa3f0146060 2024-03-25T19:36:35.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.244+0000 7fa3e6ffd640 10 client.4648.objecter ms_dispatch 0x7fa3f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:35.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.244+0000 7fa3e6ffd640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:35.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.244+0000 7fa3e6ffd640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:35.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.244+0000 7fa3e6ffd640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T19:36:35.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.244+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3d804ef30 con 0x7fa3f0146060 2024-03-25T19:36:35.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.245+0000 7fa3e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:35.244 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.245+0000 7fa3e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:35.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.471+0000 7fa3f4f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:35.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.471+0000 7fa3f4f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:35.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.471+0000 7fa3f4f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:35.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.471+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/3919897408 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3f00a9ac0 con 0x7fa3f0146060 2024-03-25T19:36:35.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.510+0000 7fa3e6ffd640 1 -- 172.21.15.52:0/3919897408 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fa3d8051030 con 0x7fa3f0146060 2024-03-25T19:36:35.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.510+0000 7fa3e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:35.510 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.510+0000 7fa3e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:35.510 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:36:35.510 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:35.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.513+0000 7fa3f4f9d640 10 librados: watch_flush enter 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.513+0000 7fa3f4f9d640 10 librados: watch_flush exit 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.513+0000 7fa3f4f9d640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.513+0000 7fa3f4f9d640 10 client.4648.objecter successfully canceled tick 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.513+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/3919897408 >> v1:172.21.15.52:6800/3967427147 conn(0x7fa3d006e010 legacy=0x7fa3d00704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.513+0000 7fa3f4f9d640 10 monclient: shutdown 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.513+0000 7fa3f4f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.513+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/3919897408 >> v1:172.21.15.179:6789/0 conn(0x7fa3f0146060 legacy=0x7fa3f01db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.514+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/3919897408 shutdown_connections 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.514+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/3919897408 >> 172.21.15.52:0/3919897408 conn(0x7fa3f00b7ea0 msgr2=0x7fa3f013dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.514+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/3919897408 shutdown_connections 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.514+0000 7fa3f4f9d640 1 -- 172.21.15.52:0/3919897408 wait complete. 2024-03-25T19:36:35.514 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:35.514+0000 7fa3f4f9d640 1 librados: shutdown 2024-03-25T19:36:35.524 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:37.846 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:09:31.445 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.852 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:09:31.450 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.880 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:09:31.479 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.902 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:09:31.444 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.908 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:09:31.450 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.913 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:09:31.510 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.928 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:09:31.527 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.951 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:09:31.493 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:37.967 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:09:31.566 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:42.331 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:42.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.448+0000 7f63cbe14640 10 monclient: get_monmap_and_config 2024-03-25T19:36:42.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.448+0000 7f63cbe14640 10 monclient: build_initial_monmap 2024-03-25T19:36:42.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.448+0000 7f63cbe14640 10 monclient: monmap: 2024-03-25T19:36:42.448 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:42.448 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:42.448 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:36:42.449278+0000 2024-03-25T19:36:42.448 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:36:42.449278+0000 2024-03-25T19:36:42.448 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:42.449 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:42.449 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:42.449 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:36:42.449 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:36:42.449 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:42.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.449+0000 7f63cbe14640 1 Processor -- start 2024-03-25T19:36:42.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.450+0000 7f63cbe14640 1 -- start start 2024-03-25T19:36:42.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.450+0000 7f63cbe14640 10 monclient: init 2024-03-25T19:36:42.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.451+0000 7f63cbe14640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:42.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.451+0000 7f63cbe14640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:42.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.451+0000 7f63cbe14640 10 monclient(hunting): picked mon.noname-a con 0x7f63c40a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:42.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.451+0000 7f63cbe14640 10 monclient(hunting): picked mon.noname-c con 0x7f63c413f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:42.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.451+0000 7f63cbe14640 10 monclient(hunting): picked mon.noname-b con 0x7f63c41430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:42.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.451+0000 7f63cbe14640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c4142520 con 0x7f63c40a98f0 2024-03-25T19:36:42.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.451+0000 7f63cbe14640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40b28a0 con 0x7f63c413f710 2024-03-25T19:36:42.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.451+0000 7f63cbe14640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40b2a30 con 0x7f63c41430c0 2024-03-25T19:36:42.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.451+0000 7f63cbe14640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:42.451 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.451+0000 7f63cbe14640 10 monclient(hunting): authenticate will time out at 6450.631193s 2024-03-25T19:36:42.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.460+0000 7f63c9bb0640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f63c40a98f0 0x7f63c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:34468/0 (socket says 172.21.15.52:34468) 2024-03-25T19:36:42.460 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.460+0000 7f63c9bb0640 1 -- 172.21.15.52:0/1462695834 learned_addr learned my addr 172.21.15.52:0/1462695834 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:42.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.466+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927675851 0 0) 0x7f63c4142520 con 0x7f63c40a98f0 2024-03-25T19:36:42.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.466+0000 7f63c8bae640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.466+0000 7f63c8bae640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.466+0000 7f63c8bae640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.465 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.466+0000 7f63c8bae640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:36:42.466 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.466+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b0003520 con 0x7f63c40a98f0 2024-03-25T19:36:42.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.470+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455242675 0 0) 0x7f63c40b28a0 con 0x7f63c413f710 2024-03-25T19:36:42.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.470+0000 7f63c8bae640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.470+0000 7f63c8bae640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.470+0000 7f63c8bae640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.470+0000 7f63c8bae640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:36:42.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.470+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c4142520 con 0x7f63c413f710 2024-03-25T19:36:42.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.471+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601409142 0 0) 0x7f63c40b2a30 con 0x7f63c41430c0 2024-03-25T19:36:42.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.471+0000 7f63c8bae640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.471+0000 7f63c8bae640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.471+0000 7f63c8bae640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.471+0000 7f63c8bae640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:36:42.471 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.471+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c40b28a0 con 0x7f63c41430c0 2024-03-25T19:36:42.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2467081743 0 0) 0x7f63b0003520 con 0x7f63c40a98f0 2024-03-25T19:36:42.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 >> v1:172.21.15.52:6790/0 conn(0x7f63c41430c0 legacy=0x7f63c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 >> v1:172.21.15.179:6789/0 conn(0x7f63c413f710 legacy=0x7f63c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 10 monclient: _finish_hunting 0 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 1 monclient: found mon.noname-a 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c40b44c0 con 0x7f63c40a98f0 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 10 monclient: _finish_auth 0 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 10 monclient: _check_auth_tickets 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f63b4001690 con 0x7f63c40a98f0 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63c8bae640 10 monclient: dump: 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:42.474 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.475 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:42.475 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:42.475 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:42.475 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:42.475 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:42.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63cbe14640 5 monclient: authenticate success, global_id 4638 2024-03-25T19:36:42.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.474+0000 7f63cbe14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.487+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63b40017f0 con 0x7f63c40a98f0 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.487+0000 7f63c8bae640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.487+0000 7f63c8bae640 1 -- 172.21.15.52:0/1462695834 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f63b4001c00 con 0x7f63c40a98f0 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.487+0000 7f63cbe14640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.487+0000 7f63cbe14640 4 set_mon_vals no callback set 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.487+0000 7f63c8bae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.487+0000 7f63c8bae640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.487+0000 7f63c8bae640 10 monclient: dump: 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:42.487 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:42.488 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.488+0000 7f63cbe14640 10 monclient: shutdown 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.488+0000 7f63cbe14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.488+0000 7f63cbe14640 1 -- 172.21.15.52:0/1462695834 >> v1:172.21.15.52:6789/0 conn(0x7f63c40a98f0 legacy=0x7f63c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.488+0000 7f63cbe14640 1 -- 172.21.15.52:0/1462695834 shutdown_connections 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.488+0000 7f63cbe14640 1 -- 172.21.15.52:0/1462695834 >> 172.21.15.52:0/1462695834 conn(0x7f63c40b7ea0 msgr2=0x7f63c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.488+0000 7f63cbe14640 1 -- 172.21.15.52:0/1462695834 shutdown_connections 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.488+0000 7f63cbe14640 1 -- 172.21.15.52:0/1462695834 wait complete. 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.489+0000 7f63cbe14640 10 monclient: build_initial_monmap 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.489+0000 7f63cbe14640 10 monclient: monmap: 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:42.490 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.489+0000 7f63cbe14640 1 librados: starting msgr at 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.489+0000 7f63cbe14640 1 librados: starting objecter 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.489+0000 7f63cbe14640 1 Processor -- start 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.489+0000 7f63cbe14640 1 -- start start 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.489+0000 7f63cbe14640 1 librados: setting wanted keys 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.489+0000 7f63cbe14640 1 librados: calling monclient init 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.489+0000 7f63cbe14640 10 monclient: init 2024-03-25T19:36:42.491 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.490+0000 7f63cbe14640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:42.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.490+0000 7f63cbe14640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:42.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.490+0000 7f63cbe14640 10 monclient(hunting): picked mon.noname-a con 0x7f63c40a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:42.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.490+0000 7f63cbe14640 10 monclient(hunting): picked mon.noname-c con 0x7f63c413f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:42.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.490+0000 7f63cbe14640 10 monclient(hunting): picked mon.noname-b con 0x7f63c41430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:42.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.490+0000 7f63cbe14640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c41db050 con 0x7f63c40a98f0 2024-03-25T19:36:42.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.490+0000 7f63cbe14640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c41dab70 con 0x7f63c41430c0 2024-03-25T19:36:42.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.490+0000 7f63cbe14640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c41dad00 con 0x7f63c413f710 2024-03-25T19:36:42.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.490+0000 7f63cbe14640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:42.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.490+0000 7f63cbe14640 10 monclient(hunting): authenticate will time out at 6450.670463s 2024-03-25T19:36:42.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.507+0000 7f63c9bb0640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f63c40a98f0 0x7f63c41db4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:34514/0 (socket says 172.21.15.52:34514) 2024-03-25T19:36:42.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.507+0000 7f63c9bb0640 1 -- 172.21.15.52:0/701790124 learned_addr learned my addr 172.21.15.52:0/701790124 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:42.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.508+0000 7f63ba7fc640 10 client.?.objecter ms_handle_connect 0x7f63c40a98f0 2024-03-25T19:36:42.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.508+0000 7f63ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:42.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.513+0000 7f63ba7fc640 10 client.?.objecter ms_handle_connect 0x7f63c413f710 2024-03-25T19:36:42.513 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.513+0000 7f63ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:42.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.515+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676066556 0 0) 0x7f63c41db050 con 0x7f63c40a98f0 2024-03-25T19:36:42.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.515+0000 7f63ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.515+0000 7f63ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.515+0000 7f63ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.515+0000 7f63ba7fc640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:36:42.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.515+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a8004eb0 con 0x7f63c40a98f0 2024-03-25T19:36:42.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.516+0000 7f63ba7fc640 10 client.?.objecter ms_handle_connect 0x7f63c41430c0 2024-03-25T19:36:42.515 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.516+0000 7f63ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.525+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1883792891 0 0) 0x7f63a8004eb0 con 0x7f63c40a98f0 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.525+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63c41db050 con 0x7f63c40a98f0 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.525+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f63b4001bc0 con 0x7f63c40a98f0 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.525+0000 7f63ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.525+0000 7f63ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.525+0000 7f63ba7fc640 10 monclient(hunting): dump: 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.525 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:42.526 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:42.526 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:42.526 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:42.526 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:42.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.525+0000 7f63ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:42.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.526+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497440307 0 0) 0x7f63c41dad00 con 0x7f63c413f710 2024-03-25T19:36:42.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.526+0000 7f63ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:42.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.526+0000 7f63ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:42.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.526+0000 7f63ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:42.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.526+0000 7f63ba7fc640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:36:42.527 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.526+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a8004eb0 con 0x7f63c413f710 2024-03-25T19:36:42.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.533+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3363273717 0 0) 0x7f63c41db050 con 0x7f63c40a98f0 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.533+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 >> v1:172.21.15.52:6790/0 conn(0x7f63c413f710 legacy=0x7f63c41d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.533+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 >> v1:172.21.15.179:6789/0 conn(0x7f63c41430c0 legacy=0x7f63c41393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.533+0000 7f63ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.533+0000 7f63ba7fc640 1 monclient: found mon.a 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.533+0000 7f63ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.533+0000 7f63ba7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.533+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c40b70a0 con 0x7f63c40a98f0 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.533+0000 7f63ba7fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.533+0000 7f63ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.533+0000 7f63ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.534+0000 7f63cbe14640 5 monclient: authenticate success, global_id 4641 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.534+0000 7f63cbe14640 10 monclient: _renew_subs 2024-03-25T19:36:42.534 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.534+0000 7f63cbe14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:36:42.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.534+0000 7f63cbe14640 1 -- 172.21.15.52:0/701790124 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c40b72f0 con 0x7f63c40a98f0 2024-03-25T19:36:42.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.534+0000 7f63cbe14640 10 client.4641.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:42.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.534+0000 7f63cbe14640 10 monclient: _renew_subs 2024-03-25T19:36:42.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.534+0000 7f63cbe14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:36:42.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.534+0000 7f63cbe14640 1 -- 172.21.15.52:0/701790124 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c41e32a0 con 0x7f63c40a98f0 2024-03-25T19:36:42.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.534+0000 7f63cbe14640 1 librados: init done 2024-03-25T19:36:42.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.535+0000 7f63cbe14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:42.535 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.535+0000 7f63cbe14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:42.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.535+0000 7f63cbe14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:36:42.536 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.535+0000 7f63cbe14640 1 -- 172.21.15.52:0/701790124 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c40627b0 con 0x7f63c40a98f0 2024-03-25T19:36:42.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.851+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63b40011f0 con 0x7f63c40a98f0 2024-03-25T19:36:42.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.851+0000 7f63ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:42.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.851+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f63b4004c70 con 0x7f63c40a98f0 2024-03-25T19:36:42.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.851+0000 7f63ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:42.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.851+0000 7f63ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:42.851 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.851+0000 7f63ba7fc640 10 monclient: dump: 2024-03-25T19:36:42.851 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:42.851 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:42.851 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:42.851 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:42.851 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:42.852 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:42.852 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:42.852 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:42.852 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:42.852 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:42.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.851+0000 7f63bbfff640 4 set_mon_vals no callback set 2024-03-25T19:36:42.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.852+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f63b4004e90 con 0x7f63c40a98f0 2024-03-25T19:36:42.852 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.853+0000 7f63c93af640 10 monclient: get_auth_request con 0x7f63a806dd40 auth_method 0 2024-03-25T19:36:42.853 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.853+0000 7f63ba7fc640 10 client.4641.objecter ms_handle_connect 0x7f63a806dd40 2024-03-25T19:36:42.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.859+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f63b4054910 con 0x7f63c40a98f0 2024-03-25T19:36:42.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.859+0000 7f63ba7fc640 10 client.4641.objecter ms_dispatch 0x7f63c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:42.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.859+0000 7f63ba7fc640 3 client.4641.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:42.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.859+0000 7f63ba7fc640 3 client.4641.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:42.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.859+0000 7f63ba7fc640 20 client.4641.objecter dump_active .. 0 homeless 2024-03-25T19:36:42.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.859+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63b4050250 con 0x7f63c40a98f0 2024-03-25T19:36:42.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.859+0000 7f63ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:42.859 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:42.859+0000 7f63ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:43.125 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.126+0000 7f63cbe14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.126+0000 7f63cbe14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.126+0000 7f63cbe14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:36:43.126 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.126+0000 7f63cbe14640 1 -- 172.21.15.52:0/701790124 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63c4069f40 con 0x7f63c40a98f0 2024-03-25T19:36:43.145 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.146+0000 7f63ba7fc640 1 -- 172.21.15.52:0/701790124 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f63b4052350 con 0x7f63c40a98f0 2024-03-25T19:36:43.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.146+0000 7f63ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:43.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.146+0000 7f63ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:43.146 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:36:43.146 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:43.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 10 librados: watch_flush enter 2024-03-25T19:36:43.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 10 librados: watch_flush exit 2024-03-25T19:36:43.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 20 client.4641.objecter shutdown clearing up homeless session... 2024-03-25T19:36:43.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 10 client.4641.objecter successfully canceled tick 2024-03-25T19:36:43.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 1 -- 172.21.15.52:0/701790124 >> v1:172.21.15.52:6800/3967427147 conn(0x7f63a806dd40 legacy=0x7f63a8070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 10 monclient: shutdown 2024-03-25T19:36:43.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:43.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 1 -- 172.21.15.52:0/701790124 >> v1:172.21.15.52:6789/0 conn(0x7f63c40a98f0 legacy=0x7f63c41db4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:43.148 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 1 -- 172.21.15.52:0/701790124 shutdown_connections 2024-03-25T19:36:43.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 1 -- 172.21.15.52:0/701790124 >> 172.21.15.52:0/701790124 conn(0x7f63c40b7ea0 msgr2=0x7f63c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:43.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 1 -- 172.21.15.52:0/701790124 shutdown_connections 2024-03-25T19:36:43.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 1 -- 172.21.15.52:0/701790124 wait complete. 2024-03-25T19:36:43.149 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:43.148+0000 7f63cbe14640 1 librados: shutdown 2024-03-25T19:36:43.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:49.966 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:50.083 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.084+0000 7f6296baf640 10 monclient: get_monmap_and_config 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.084+0000 7f6296baf640 10 monclient: build_initial_monmap 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.084+0000 7f6296baf640 10 monclient: monmap: 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:36:50.084610+0000 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:36:50.084610+0000 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:36:50.084 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:50.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.085+0000 7f6296baf640 1 Processor -- start 2024-03-25T19:36:50.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.085+0000 7f6296baf640 1 -- start start 2024-03-25T19:36:50.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.085+0000 7f6296baf640 10 monclient: init 2024-03-25T19:36:50.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.086+0000 7f6296baf640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:50.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.086+0000 7f6296baf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:50.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.086+0000 7f6296baf640 10 monclient(hunting): picked mon.noname-a con 0x7f62900b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:50.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.086+0000 7f6296baf640 10 monclient(hunting): picked mon.noname-b con 0x7f62900b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.086+0000 7f6296baf640 10 monclient(hunting): picked mon.noname-c con 0x7f629014be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.086+0000 7f6296baf640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62900af940 con 0x7f62900b4d60 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.086+0000 7f6296baf640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62900b0a70 con 0x7f629014be90 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.086+0000 7f6296baf640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62900b0d50 con 0x7f62900b1730 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.086+0000 7f6296baf640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:50.087 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.086+0000 7f6296baf640 10 monclient(hunting): authenticate will time out at 6458.266487s 2024-03-25T19:36:50.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.093+0000 7f629514c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f629014be90 0x7f629014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:57358/0 (socket says 172.21.15.52:57358) 2024-03-25T19:36:50.092 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.093+0000 7f629514c640 1 -- 172.21.15.52:0/774095256 learned_addr learned my addr 172.21.15.52:0/774095256 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:50.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.100+0000 7f628f7fe640 1 -- 172.21.15.52:0/774095256 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089843598 0 0) 0x7f62900b0a70 con 0x7f629014be90 2024-03-25T19:36:50.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.100+0000 7f628f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.100+0000 7f628f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.100+0000 7f628f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.100+0000 7f628f7fe640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:36:50.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.100+0000 7f628f7fe640 1 -- 172.21.15.52:0/774095256 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6280003520 con 0x7f629014be90 2024-03-25T19:36:50.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.112+0000 7f628f7fe640 1 -- 172.21.15.52:0/774095256 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887203623 0 0) 0x7f62900b0d50 con 0x7f62900b1730 2024-03-25T19:36:50.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.112+0000 7f628f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.111 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.112+0000 7f628f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.112+0000 7f628f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.112+0000 7f628f7fe640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:36:50.112 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.112+0000 7f628f7fe640 1 -- 172.21.15.52:0/774095256 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62900b0a70 con 0x7f62900b1730 2024-03-25T19:36:50.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 1 -- 172.21.15.52:0/774095256 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 492939501 0 0) 0x7f6280003520 con 0x7f629014be90 2024-03-25T19:36:50.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 1 -- 172.21.15.52:0/774095256 >> v1:172.21.15.52:6790/0 conn(0x7f62900b1730 legacy=0x7f62900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 1 -- 172.21.15.52:0/774095256 >> v1:172.21.15.52:6789/0 conn(0x7f62900b4d60 legacy=0x7f629013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:36:50.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:50.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:36:50.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:50.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:36:50.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 1 -- 172.21.15.52:0/774095256 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62900b1030 con 0x7f629014be90 2024-03-25T19:36:50.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:50.117 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 1 -- 172.21.15.52:0/774095256 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6288001a10 con 0x7f629014be90 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f628f7fe640 10 monclient: dump: 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:50.118 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:50.119 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:50.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f6296baf640 5 monclient: authenticate success, global_id 4657 2024-03-25T19:36:50.119 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.117+0000 7f6296baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:50.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.128+0000 7f628f7fe640 1 -- 172.21.15.52:0/774095256 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6288001b70 con 0x7f629014be90 2024-03-25T19:36:50.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.128+0000 7f628f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:50.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.128+0000 7f628f7fe640 1 -- 172.21.15.52:0/774095256 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6288001f80 con 0x7f629014be90 2024-03-25T19:36:50.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.128+0000 7f6296baf640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:50.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.128+0000 7f6296baf640 4 set_mon_vals no callback set 2024-03-25T19:36:50.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.128+0000 7f628f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.128+0000 7f628f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:50.128 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.128+0000 7f628f7fe640 10 monclient: dump: 2024-03-25T19:36:50.128 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:50.128 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:50.128 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:50.129 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:50.129 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.129 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:50.129 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:50.129 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:50.129 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:50.129 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:50.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.129+0000 7f6296baf640 10 monclient: shutdown 2024-03-25T19:36:50.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.129+0000 7f6296baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:50.131 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.129+0000 7f6296baf640 1 -- 172.21.15.52:0/774095256 >> v1:172.21.15.179:6789/0 conn(0x7f629014be90 legacy=0x7f629014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.129+0000 7f6296baf640 1 -- 172.21.15.52:0/774095256 shutdown_connections 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.129+0000 7f6296baf640 1 -- 172.21.15.52:0/774095256 >> 172.21.15.52:0/774095256 conn(0x7f62900617a0 msgr2=0x7f6290061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.129+0000 7f6296baf640 1 -- 172.21.15.52:0/774095256 shutdown_connections 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.129+0000 7f6296baf640 1 -- 172.21.15.52:0/774095256 wait complete. 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.130+0000 7f6296baf640 10 monclient: build_initial_monmap 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.130+0000 7f6296baf640 10 monclient: monmap: 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:36:50.132 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.130+0000 7f6296baf640 1 librados: starting msgr at 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.130+0000 7f6296baf640 1 librados: starting objecter 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.130+0000 7f6296baf640 1 Processor -- start 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 1 -- start start 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 1 librados: setting wanted keys 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 1 librados: calling monclient init 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 10 monclient: init 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 10 monclient(hunting): picked mon.noname-b con 0x7f62900b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 10 monclient(hunting): picked mon.noname-c con 0x7f62900b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 10 monclient(hunting): picked mon.noname-a con 0x7f629014be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62901410d0 con 0x7f629014be90 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6290143690 con 0x7f62900b1730 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62901415d0 con 0x7f62900b4d60 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:50.133 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.131+0000 7f6296baf640 10 monclient(hunting): authenticate will time out at 6458.311656s 2024-03-25T19:36:50.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.139+0000 7f629494b640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f62900b1730 0x7f6290140890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:57414/0 (socket says 172.21.15.52:57414) 2024-03-25T19:36:50.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.139+0000 7f629494b640 1 -- 172.21.15.52:0/2658758225 learned_addr learned my addr 172.21.15.52:0/2658758225 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:50.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.140+0000 7f628d7fa640 10 client.?.objecter ms_handle_connect 0x7f62900b1730 2024-03-25T19:36:50.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.140+0000 7f628d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.144+0000 7f628d7fa640 10 client.?.objecter ms_handle_connect 0x7f629014be90 2024-03-25T19:36:50.143 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.144+0000 7f628d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.146+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193185087 0 0) 0x7f6290143690 con 0x7f62900b1730 2024-03-25T19:36:50.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.146+0000 7f628d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.146+0000 7f628d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.146+0000 7f628d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.146+0000 7f628d7fa640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:36:50.146 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.146+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6274004eb0 con 0x7f62900b1730 2024-03-25T19:36:50.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.151+0000 7f628d7fa640 10 client.?.objecter ms_handle_connect 0x7f62900b4d60 2024-03-25T19:36:50.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.151+0000 7f628d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:50.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.151+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005608011 0 0) 0x7f62901410d0 con 0x7f629014be90 2024-03-25T19:36:50.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.151+0000 7f628d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:50.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.151+0000 7f628d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:50.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.151+0000 7f628d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:50.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.151+0000 7f628d7fa640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:36:50.151 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.151+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6290143690 con 0x7f629014be90 2024-03-25T19:36:50.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.158+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3368125364 0 0) 0x7f6274004eb0 con 0x7f62900b1730 2024-03-25T19:36:50.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.158+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62901410d0 con 0x7f62900b1730 2024-03-25T19:36:50.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.158+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6284001960 con 0x7f62900b1730 2024-03-25T19:36:50.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.158+0000 7f628d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.158+0000 7f628d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:50.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.158+0000 7f628d7fa640 10 monclient(hunting): dump: 2024-03-25T19:36:50.158 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:50.158 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:50.159 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:50.159 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:50.159 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.159 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:50.159 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:50.159 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:50.159 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:50.159 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:50.159 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.158+0000 7f628d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:50.167 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.167+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4263043848 0 0) 0x7f6290143690 con 0x7f629014be90 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.168+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6274004eb0 con 0x7f629014be90 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.168+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6288001e80 con 0x7f629014be90 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.168+0000 7f628d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.168+0000 7f628d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.168+0000 7f628d7fa640 10 monclient(hunting): dump: 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:50.168 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:50.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1574145124 0 0) 0x7f62901410d0 con 0x7f62900b1730 2024-03-25T19:36:50.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 >> v1:172.21.15.52:6790/0 conn(0x7f62900b4d60 legacy=0x7f6290142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 >> v1:172.21.15.52:6789/0 conn(0x7f629014be90 legacy=0x7f6290136160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f628d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:36:50.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f628d7fa640 1 monclient: found mon.b 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f628d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f628d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6290136870 con 0x7f62900b1730 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f628d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f628d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f628d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f6296baf640 5 monclient: authenticate success, global_id 4660 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f6296baf640 10 monclient: _renew_subs 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f6296baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f6296baf640 1 -- 172.21.15.52:0/2658758225 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6290136b40 con 0x7f62900b1730 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f6296baf640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f6296baf640 10 monclient: _renew_subs 2024-03-25T19:36:50.174 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f6296baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:50.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f6296baf640 1 -- 172.21.15.52:0/2658758225 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62901e3630 con 0x7f62900b1730 2024-03-25T19:36:50.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.173+0000 7f6296baf640 1 librados: init done 2024-03-25T19:36:50.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.175+0000 7f6296baf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.175+0000 7f6296baf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.175+0000 7f6296baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:50.175 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.175+0000 7f6296baf640 1 -- 172.21.15.52:0/2658758225 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62900627b0 con 0x7f62900b1730 2024-03-25T19:36:50.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.492+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62840035f0 con 0x7f62900b1730 2024-03-25T19:36:50.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.492+0000 7f628d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:50.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.492+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6284003940 con 0x7f62900b1730 2024-03-25T19:36:50.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.492+0000 7f628d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:50.492 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.492+0000 7f628d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.492+0000 7f628d7fa640 10 monclient: dump: 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:50.493 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.492+0000 7f628effd640 4 set_mon_vals no callback set 2024-03-25T19:36:50.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.493+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f6284003b60 con 0x7f62900b1730 2024-03-25T19:36:50.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.494+0000 7f628ffff640 10 monclient: get_auth_request con 0x7f627406de90 auth_method 0 2024-03-25T19:36:50.494 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.494+0000 7f628d7fa640 10 client.4660.objecter ms_handle_connect 0x7f627406de90 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.499+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f6284054750 con 0x7f62900b1730 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.499+0000 7f628d7fa640 10 client.4660.objecter ms_dispatch 0x7f6290000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.499+0000 7f628d7fa640 3 client.4660.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.499+0000 7f628d7fa640 3 client.4660.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:50.499 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.499+0000 7f628d7fa640 20 client.4660.objecter dump_active .. 0 homeless 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.500+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6284050260 con 0x7f62900b1730 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.500+0000 7f628d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:50.500 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.500+0000 7f628d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:50.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.731+0000 7f6296baf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:50.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.731+0000 7f6296baf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:50.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.731+0000 7f6296baf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:36:50.731 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.731+0000 7f6296baf640 1 -- 172.21.15.52:0/2658758225 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62900b5f40 con 0x7f62900b1730 2024-03-25T19:36:50.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.769+0000 7f628d7fa640 1 -- 172.21.15.52:0/2658758225 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f6284052190 con 0x7f62900b1730 2024-03-25T19:36:50.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.769+0000 7f628d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:50.770 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.769+0000 7f628d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:50.770 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:36:50.770 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:50.772 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 10 librados: watch_flush enter 2024-03-25T19:36:50.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 10 librados: watch_flush exit 2024-03-25T19:36:50.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 20 client.4660.objecter shutdown clearing up homeless session... 2024-03-25T19:36:50.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 10 client.4660.objecter successfully canceled tick 2024-03-25T19:36:50.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 1 -- 172.21.15.52:0/2658758225 >> v1:172.21.15.52:6800/3967427147 conn(0x7f627406de90 legacy=0x7f6274070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 10 monclient: shutdown 2024-03-25T19:36:50.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:50.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 1 -- 172.21.15.52:0/2658758225 >> v1:172.21.15.179:6789/0 conn(0x7f62900b1730 legacy=0x7f6290140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:50.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 1 -- 172.21.15.52:0/2658758225 shutdown_connections 2024-03-25T19:36:50.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 1 -- 172.21.15.52:0/2658758225 >> 172.21.15.52:0/2658758225 conn(0x7f62900617a0 msgr2=0x7f629013c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:50.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 1 -- 172.21.15.52:0/2658758225 shutdown_connections 2024-03-25T19:36:50.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 1 -- 172.21.15.52:0/2658758225 wait complete. 2024-03-25T19:36:50.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:50.772+0000 7f6296baf640 1 librados: shutdown 2024-03-25T19:36:50.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:57.591 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:57.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.709+0000 7fc693132640 10 monclient: get_monmap_and_config 2024-03-25T19:36:57.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.709+0000 7fc693132640 10 monclient: build_initial_monmap 2024-03-25T19:36:57.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.709+0000 7fc693132640 10 monclient: monmap: 2024-03-25T19:36:57.709 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:57.709 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:57.709 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:36:57.710490+0000 2024-03-25T19:36:57.709 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:36:57.710490+0000 2024-03-25T19:36:57.710 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:57.710 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:57.710 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:57.710 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:36:57.710 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:36:57.710 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:57.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.710+0000 7fc693132640 1 Processor -- start 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.710+0000 7fc693132640 1 -- start start 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.710+0000 7fc693132640 10 monclient: init 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.711+0000 7fc693132640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.711+0000 7fc693132640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.711+0000 7fc693132640 10 monclient(hunting): picked mon.noname-a con 0x7fc68c13fc60 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.711+0000 7fc693132640 10 monclient(hunting): picked mon.noname-b con 0x7fc68c070570 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.711+0000 7fc693132640 10 monclient(hunting): picked mon.noname-c con 0x7fc68c0727a0 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.711+0000 7fc693132640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68c13c9f0 con 0x7fc68c13fc60 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.711+0000 7fc693132640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68c06fac0 con 0x7fc68c0727a0 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.711+0000 7fc693132640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68c0715d0 con 0x7fc68c070570 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.711+0000 7fc693132640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:57.713 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.711+0000 7fc693132640 10 monclient(hunting): authenticate will time out at 6465.891479s 2024-03-25T19:36:57.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.713+0000 7fc68bfff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fc68c070570 0x7fc68c072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:50094/0 (socket says 172.21.15.52:50094) 2024-03-25T19:36:57.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.713+0000 7fc68bfff640 1 -- 172.21.15.52:0/3628375824 learned_addr learned my addr 172.21.15.52:0/3628375824 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:57.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.717+0000 7fc68b7fe640 1 -- 172.21.15.52:0/3628375824 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166939675 0 0) 0x7fc68c0715d0 con 0x7fc68c070570 2024-03-25T19:36:57.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.717+0000 7fc68b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:36:57.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.717+0000 7fc68b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:57.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.717+0000 7fc68b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:57.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.717+0000 7fc68b7fe640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:36:57.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.717+0000 7fc68b7fe640 1 -- 172.21.15.52:0/3628375824 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc67c005730 con 0x7fc68c070570 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 1 -- 172.21.15.52:0/3628375824 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1761444729 0 0) 0x7fc67c005730 con 0x7fc68c070570 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 1 -- 172.21.15.52:0/3628375824 >> v1:172.21.15.179:6789/0 conn(0x7fc68c0727a0 legacy=0x7fc68c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 1 -- 172.21.15.52:0/3628375824 >> v1:172.21.15.52:6789/0 conn(0x7fc68c13fc60 legacy=0x7fc68c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 1 monclient: found mon.noname-b 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 1 -- 172.21.15.52:0/3628375824 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c0718b0 con 0x7fc68c070570 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:57.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 1 -- 172.21.15.52:0/3628375824 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc678001690 con 0x7fc68c070570 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc693132640 5 monclient: authenticate success, global_id 4643 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc693132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc68b7fe640 10 monclient: dump: 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:57.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.727+0000 7fc693132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:57.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.738+0000 7fc68b7fe640 1 -- 172.21.15.52:0/3628375824 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc678001920 con 0x7fc68c070570 2024-03-25T19:36:57.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.738+0000 7fc68b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:57.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.738+0000 7fc68b7fe640 1 -- 172.21.15.52:0/3628375824 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc678001d30 con 0x7fc68c070570 2024-03-25T19:36:57.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.738+0000 7fc68b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:57.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.738+0000 7fc68b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:57.738 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.738+0000 7fc68b7fe640 10 monclient: dump: 2024-03-25T19:36:57.738 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:57.738 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:57.738 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:57.738 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:57.739 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:57.739 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:57.739 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:57.739 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:57.739 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:57.739 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:57.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.738+0000 7fc693132640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:57.739 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.738+0000 7fc693132640 4 set_mon_vals no callback set 2024-03-25T19:36:57.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.739+0000 7fc693132640 10 monclient: shutdown 2024-03-25T19:36:57.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.739+0000 7fc693132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:57.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.739+0000 7fc693132640 1 -- 172.21.15.52:0/3628375824 >> v1:172.21.15.52:6790/0 conn(0x7fc68c070570 legacy=0x7fc68c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:57.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.739+0000 7fc693132640 1 -- 172.21.15.52:0/3628375824 shutdown_connections 2024-03-25T19:36:57.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.739+0000 7fc693132640 1 -- 172.21.15.52:0/3628375824 >> 172.21.15.52:0/3628375824 conn(0x7fc68c0b4a80 msgr2=0x7fc68c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:57.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.739+0000 7fc693132640 1 -- 172.21.15.52:0/3628375824 shutdown_connections 2024-03-25T19:36:57.741 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.739+0000 7fc693132640 1 -- 172.21.15.52:0/3628375824 wait complete. 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.740+0000 7fc693132640 10 monclient: build_initial_monmap 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.740+0000 7fc693132640 10 monclient: monmap: 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.740+0000 7fc693132640 1 librados: starting msgr at 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.740+0000 7fc693132640 1 librados: starting objecter 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.740+0000 7fc693132640 1 Processor -- start 2024-03-25T19:36:57.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 1 -- start start 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 1 librados: setting wanted keys 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 1 librados: calling monclient init 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 10 monclient: init 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 10 monclient(hunting): picked mon.noname-a con 0x7fc68c070570 addr v1:172.21.15.52:6789/0 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 10 monclient(hunting): picked mon.noname-c con 0x7fc68c0727a0 addr v1:172.21.15.52:6790/0 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 10 monclient(hunting): picked mon.noname-b con 0x7fc68c13fc60 addr v1:172.21.15.179:6789/0 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68c11e3d0 con 0x7fc68c070570 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68c11d280 con 0x7fc68c13fc60 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68c1d6650 con 0x7fc68c0727a0 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:57.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.741+0000 7fc693132640 10 monclient(hunting): authenticate will time out at 6465.921850s 2024-03-25T19:36:57.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.746+0000 7fc690ece640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7fc68c070570 0x7fc68c11cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:39904/0 (socket says 172.21.15.52:39904) 2024-03-25T19:36:57.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.746+0000 7fc690ece640 1 -- 172.21.15.52:0/534620180 learned_addr learned my addr 172.21.15.52:0/534620180 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:36:57.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.747+0000 7fc6897fa640 10 client.?.objecter ms_handle_connect 0x7fc68c070570 2024-03-25T19:36:57.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.747+0000 7fc6897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:57.753 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.753+0000 7fc6897fa640 10 client.?.objecter ms_handle_connect 0x7fc68c0727a0 2024-03-25T19:36:57.753 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.753+0000 7fc6897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:57.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.760+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774077410 0 0) 0x7fc68c1d6650 con 0x7fc68c0727a0 2024-03-25T19:36:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.760+0000 7fc6897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.760+0000 7fc6897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.760+0000 7fc6897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.760+0000 7fc6897fa640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:36:57.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.760+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc670004eb0 con 0x7fc68c0727a0 2024-03-25T19:36:57.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.766+0000 7fc6897fa640 10 client.?.objecter ms_handle_connect 0x7fc68c13fc60 2024-03-25T19:36:57.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.766+0000 7fc6897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:57.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.766+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246843139 0 0) 0x7fc68c11e3d0 con 0x7fc68c070570 2024-03-25T19:36:57.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.766+0000 7fc6897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:57.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.766+0000 7fc6897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:57.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.766+0000 7fc6897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:57.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.766+0000 7fc6897fa640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:36:57.766 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.766+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc68c1d6650 con 0x7fc68c070570 2024-03-25T19:36:57.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.774+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398094594 0 0) 0x7fc68c11d280 con 0x7fc68c13fc60 2024-03-25T19:36:57.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.774+0000 7fc6897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:36:57.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.774+0000 7fc6897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:57.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.774+0000 7fc6897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:57.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.774+0000 7fc6897fa640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:36:57.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.774+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc68c11e3d0 con 0x7fc68c13fc60 2024-03-25T19:36:57.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.775+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 76021573 0 0) 0x7fc670004eb0 con 0x7fc68c0727a0 2024-03-25T19:36:57.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.775+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc68c11d280 con 0x7fc68c0727a0 2024-03-25T19:36:57.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.775+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc6780014b0 con 0x7fc68c0727a0 2024-03-25T19:36:57.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.775+0000 7fc6897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:57.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.775+0000 7fc6897fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:57.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.775+0000 7fc6897fa640 10 monclient(hunting): dump: 2024-03-25T19:36:57.775 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:57.775 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:57.775 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:57.775 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:57.776 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:57.776 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:57.776 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:57.776 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:57.776 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:57.776 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:57.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.775+0000 7fc6897fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:57.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.780+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3064424218 0 0) 0x7fc68c1d6650 con 0x7fc68c070570 2024-03-25T19:36:57.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.780+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc670004eb0 con 0x7fc68c070570 2024-03-25T19:36:57.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.780+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc680001910 con 0x7fc68c070570 2024-03-25T19:36:57.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.780+0000 7fc6897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:57.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.780+0000 7fc6897fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:57.780 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.780+0000 7fc6897fa640 10 monclient(hunting): dump: 2024-03-25T19:36:57.780 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:57.781 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:57.781 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:57.781 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:57.781 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:57.781 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:57.781 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:57.781 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:57.781 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:57.781 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:57.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2588985491 0 0) 0x7fc68c11d280 con 0x7fc68c0727a0 2024-03-25T19:36:57.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 >> v1:172.21.15.179:6789/0 conn(0x7fc68c13fc60 legacy=0x7fc68c1ddac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:57.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 >> v1:172.21.15.52:6789/0 conn(0x7fc68c070570 legacy=0x7fc68c11cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:57.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc6897fa640 10 monclient: _finish_hunting 0 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc6897fa640 1 monclient: found mon.c 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc6897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc6897fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c1d7260 con 0x7fc68c0727a0 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc6897fa640 10 monclient: _finish_auth 0 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc6897fa640 10 monclient: _check_auth_tickets 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc6897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc693132640 5 monclient: authenticate success, global_id 4646 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc693132640 10 monclient: _renew_subs 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc693132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc693132640 1 -- 172.21.15.52:0/534620180 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc68c0ab620 con 0x7fc68c0727a0 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc693132640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc693132640 10 monclient: _renew_subs 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc693132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:36:57.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc693132640 1 -- 172.21.15.52:0/534620180 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68c0ae400 con 0x7fc68c0727a0 2024-03-25T19:36:57.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.786+0000 7fc693132640 1 librados: init done 2024-03-25T19:36:57.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.787+0000 7fc693132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:57.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.787+0000 7fc693132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:57.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.787+0000 7fc693132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:36:57.788 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:57.787+0000 7fc693132640 1 -- 172.21.15.52:0/534620180 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68c1d67e0 con 0x7fc68c0727a0 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.099+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6780016d0 con 0x7fc68c0727a0 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.099+0000 7fc6897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.099+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fc678004dd0 con 0x7fc68c0727a0 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.099+0000 7fc6897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.099+0000 7fc6897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.099+0000 7fc6897fa640 10 monclient: dump: 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:36:58.099 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:58.100 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:36:58.100 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:58.100 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:36:58.100 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:36:58.100 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:36:58.100 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:36:58.100 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:36:58.100 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.099+0000 7fc68affd640 4 set_mon_vals no callback set 2024-03-25T19:36:58.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.100+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fc67801ada0 con 0x7fc68c0727a0 2024-03-25T19:36:58.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.100+0000 7fc690ece640 10 monclient: get_auth_request con 0x7fc67006df70 auth_method 0 2024-03-25T19:36:58.101 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.100+0000 7fc6897fa640 10 client.4646.objecter ms_handle_connect 0x7fc67006df70 2024-03-25T19:36:58.107 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.108+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fc6780548b0 con 0x7fc68c0727a0 2024-03-25T19:36:58.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.108+0000 7fc6897fa640 10 client.4646.objecter ms_dispatch 0x7fc68c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:58.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.108+0000 7fc6897fa640 3 client.4646.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:58.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.108+0000 7fc6897fa640 3 client.4646.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:58.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.108+0000 7fc6897fa640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-25T19:36:58.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.108+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6780503c0 con 0x7fc68c0727a0 2024-03-25T19:36:58.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.108+0000 7fc6897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:58.108 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.108+0000 7fc6897fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:58.188 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:09:51.730 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.192 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:09:51.734 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.195 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:09:51.794 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.200 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:09:51.799 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.233 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:09:51.775 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.235 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:09:51.834 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.275 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:09:51.874 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.292 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:09:51.890 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.319 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:09:51.918 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:58.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.369+0000 7fc693132640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:58.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.369+0000 7fc693132640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:58.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.369+0000 7fc693132640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:36:58.368 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.369+0000 7fc693132640 1 -- 172.21.15.52:0/534620180 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc68c1d73f0 con 0x7fc68c0727a0 2024-03-25T19:36:58.582 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.583+0000 7fc6897fa640 1 -- 172.21.15.52:0/534620180 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fc68c1d73f0 con 0x7fc68c0727a0 2024-03-25T19:36:58.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.583+0000 7fc6897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:58.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.583+0000 7fc6897fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:58.583 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:36:58.583 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:58.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.585+0000 7fc693132640 10 librados: watch_flush enter 2024-03-25T19:36:58.585 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.585+0000 7fc693132640 10 librados: watch_flush exit 2024-03-25T19:36:58.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.585+0000 7fc693132640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-25T19:36:58.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.585+0000 7fc693132640 10 client.4646.objecter successfully canceled tick 2024-03-25T19:36:58.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.585+0000 7fc693132640 1 -- 172.21.15.52:0/534620180 >> v1:172.21.15.52:6800/3967427147 conn(0x7fc67006df70 legacy=0x7fc670070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.585+0000 7fc693132640 10 monclient: shutdown 2024-03-25T19:36:58.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.585+0000 7fc693132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:58.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.585+0000 7fc693132640 1 -- 172.21.15.52:0/534620180 >> v1:172.21.15.52:6790/0 conn(0x7fc68c0727a0 legacy=0x7fc68c1d6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:58.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.585+0000 7fc693132640 1 -- 172.21.15.52:0/534620180 shutdown_connections 2024-03-25T19:36:58.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.585+0000 7fc693132640 1 -- 172.21.15.52:0/534620180 >> 172.21.15.52:0/534620180 conn(0x7fc68c0b4a80 msgr2=0x7fc68c0b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:58.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.585+0000 7fc693132640 1 -- 172.21.15.52:0/534620180 shutdown_connections 2024-03-25T19:36:58.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.586+0000 7fc693132640 1 -- 172.21.15.52:0/534620180 wait complete. 2024-03-25T19:36:58.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:36:58.586+0000 7fc693132640 1 librados: shutdown 2024-03-25T19:36:58.597 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:05.402 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.519+0000 7fd3dfed9640 10 monclient: get_monmap_and_config 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.519+0000 7fd3dfed9640 10 monclient: build_initial_monmap 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.519+0000 7fd3dfed9640 10 monclient: monmap: 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:37:05.519869+0000 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:37:05.519869+0000 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:37:05.519 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:37:05.520 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:05.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.520+0000 7fd3dfed9640 1 Processor -- start 2024-03-25T19:37:05.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.520+0000 7fd3dfed9640 1 -- start start 2024-03-25T19:37:05.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.520+0000 7fd3dfed9640 10 monclient: init 2024-03-25T19:37:05.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.521+0000 7fd3dfed9640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:05.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.521+0000 7fd3dfed9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:05.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.521+0000 7fd3dfed9640 10 monclient(hunting): picked mon.noname-a con 0x7fd3d80a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:05.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.521+0000 7fd3dfed9640 10 monclient(hunting): picked mon.noname-b con 0x7fd3d813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:05.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.521+0000 7fd3dfed9640 10 monclient(hunting): picked mon.noname-c con 0x7fd3d81430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:05.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.521+0000 7fd3dfed9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d8142520 con 0x7fd3d80a98f0 2024-03-25T19:37:05.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.521+0000 7fd3dfed9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d80b28a0 con 0x7fd3d81430c0 2024-03-25T19:37:05.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.521+0000 7fd3dfed9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d80b2a30 con 0x7fd3d813f710 2024-03-25T19:37:05.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.521+0000 7fd3dfed9640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:05.522 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.521+0000 7fd3dfed9640 10 monclient(hunting): authenticate will time out at 6473.701411s 2024-03-25T19:37:05.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.524+0000 7fd3de476640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd3d81430c0 0x7fd3d814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:35690/0 (socket says 172.21.15.52:35690) 2024-03-25T19:37:05.523 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.524+0000 7fd3de476640 1 -- 172.21.15.52:0/3873690922 learned_addr learned my addr 172.21.15.52:0/3873690922 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:05.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.532+0000 7fd3dcc73640 1 -- 172.21.15.52:0/3873690922 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872042345 0 0) 0x7fd3d80b28a0 con 0x7fd3d81430c0 2024-03-25T19:37:05.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.532+0000 7fd3dcc73640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.532+0000 7fd3dcc73640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.532+0000 7fd3dcc73640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.532+0000 7fd3dcc73640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:37:05.532 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.532+0000 7fd3dcc73640 1 -- 172.21.15.52:0/3873690922 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3c4003520 con 0x7fd3d81430c0 2024-03-25T19:37:05.548 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 1 -- 172.21.15.52:0/3873690922 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3450084802 0 0) 0x7fd3c4003520 con 0x7fd3d81430c0 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 1 -- 172.21.15.52:0/3873690922 >> v1:172.21.15.52:6790/0 conn(0x7fd3d813f710 legacy=0x7fd3d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 1 -- 172.21.15.52:0/3873690922 >> v1:172.21.15.52:6789/0 conn(0x7fd3d80a98f0 legacy=0x7fd3d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 10 monclient: _finish_hunting 0 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 1 monclient: found mon.noname-c 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 1 -- 172.21.15.52:0/3873690922 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d80b44c0 con 0x7fd3d81430c0 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 10 monclient: _finish_auth 0 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 10 monclient: _check_auth_tickets 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 1 -- 172.21.15.52:0/3873690922 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd3d0001690 con 0x7fd3d81430c0 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dfed9640 5 monclient: authenticate success, global_id 4666 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dfed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:05.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dcc73640 10 monclient: dump: 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:05.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.549+0000 7fd3dfed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:05.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.560+0000 7fd3dcc73640 1 -- 172.21.15.52:0/3873690922 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3d00017f0 con 0x7fd3d81430c0 2024-03-25T19:37:05.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.560+0000 7fd3dcc73640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:05.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.560+0000 7fd3dcc73640 1 -- 172.21.15.52:0/3873690922 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd3d0001c00 con 0x7fd3d81430c0 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.560+0000 7fd3dfed9640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.560+0000 7fd3dfed9640 4 set_mon_vals no callback set 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.560+0000 7fd3dcc73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.560+0000 7fd3dcc73640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.560+0000 7fd3dcc73640 10 monclient: dump: 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:05.560 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:05.561 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:05.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.560+0000 7fd3dfed9640 10 monclient: shutdown 2024-03-25T19:37:05.562 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.560+0000 7fd3dfed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.560+0000 7fd3dfed9640 1 -- 172.21.15.52:0/3873690922 >> v1:172.21.15.179:6789/0 conn(0x7fd3d81430c0 legacy=0x7fd3d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.561+0000 7fd3dfed9640 1 -- 172.21.15.52:0/3873690922 shutdown_connections 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.561+0000 7fd3dfed9640 1 -- 172.21.15.52:0/3873690922 >> 172.21.15.52:0/3873690922 conn(0x7fd3d80b7ea0 msgr2=0x7fd3d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.561+0000 7fd3dfed9640 1 -- 172.21.15.52:0/3873690922 shutdown_connections 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.561+0000 7fd3dfed9640 1 -- 172.21.15.52:0/3873690922 wait complete. 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.561+0000 7fd3dfed9640 10 monclient: build_initial_monmap 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.561+0000 7fd3dfed9640 10 monclient: monmap: 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:05.563 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.562+0000 7fd3dfed9640 1 librados: starting msgr at 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.562+0000 7fd3dfed9640 1 librados: starting objecter 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.562+0000 7fd3dfed9640 1 Processor -- start 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.562+0000 7fd3dfed9640 1 -- start start 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.562+0000 7fd3dfed9640 1 librados: setting wanted keys 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.562+0000 7fd3dfed9640 1 librados: calling monclient init 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.562+0000 7fd3dfed9640 10 monclient: init 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.562+0000 7fd3dfed9640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.563+0000 7fd3dfed9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.563+0000 7fd3dfed9640 10 monclient(hunting): picked mon.noname-c con 0x7fd3d80a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.563+0000 7fd3dfed9640 10 monclient(hunting): picked mon.noname-b con 0x7fd3d813f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:05.564 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.563+0000 7fd3dfed9640 10 monclient(hunting): picked mon.noname-a con 0x7fd3d81430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:05.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.563+0000 7fd3dfed9640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d81d7ba0 con 0x7fd3d81430c0 2024-03-25T19:37:05.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.563+0000 7fd3dfed9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d8138330 con 0x7fd3d813f710 2024-03-25T19:37:05.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.563+0000 7fd3dfed9640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d80ab950 con 0x7fd3d80a98f0 2024-03-25T19:37:05.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.563+0000 7fd3dfed9640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:05.565 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.563+0000 7fd3dfed9640 10 monclient(hunting): authenticate will time out at 6473.743226s 2024-03-25T19:37:05.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.576+0000 7fd3ddc75640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7fd3d80a98f0 0x7fd3d81378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:34880/0 (socket says 172.21.15.52:34880) 2024-03-25T19:37:05.576 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.576+0000 7fd3ddc75640 1 -- 172.21.15.52:0/2385919022 learned_addr learned my addr 172.21.15.52:0/2385919022 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:05.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.578+0000 7fd3ce7fc640 10 client.?.objecter ms_handle_connect 0x7fd3d813f710 2024-03-25T19:37:05.578 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.578+0000 7fd3ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:05.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.580+0000 7fd3ce7fc640 10 client.?.objecter ms_handle_connect 0x7fd3d80a98f0 2024-03-25T19:37:05.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.580+0000 7fd3ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:05.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.581+0000 7fd3ce7fc640 10 client.?.objecter ms_handle_connect 0x7fd3d81430c0 2024-03-25T19:37:05.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.581+0000 7fd3ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:05.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.589+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867179759 0 0) 0x7fd3d8138330 con 0x7fd3d813f710 2024-03-25T19:37:05.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.589+0000 7fd3ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.589+0000 7fd3ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.589+0000 7fd3ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.589+0000 7fd3ce7fc640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:37:05.589 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.589+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3bc004eb0 con 0x7fd3d813f710 2024-03-25T19:37:05.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.592+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917103126 0 0) 0x7fd3d81d7ba0 con 0x7fd3d81430c0 2024-03-25T19:37:05.592 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.593+0000 7fd3ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.593+0000 7fd3ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.593+0000 7fd3ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.593+0000 7fd3ce7fc640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:37:05.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.593+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d8138330 con 0x7fd3d81430c0 2024-03-25T19:37:05.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.593+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193305015 0 0) 0x7fd3d80ab950 con 0x7fd3d80a98f0 2024-03-25T19:37:05.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.593+0000 7fd3ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:05.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.593+0000 7fd3ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:05.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.593+0000 7fd3ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:05.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.593+0000 7fd3ce7fc640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:37:05.593 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.593+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d81d7ba0 con 0x7fd3d80a98f0 2024-03-25T19:37:05.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.623+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3456410465 0 0) 0x7fd3d8138330 con 0x7fd3d81430c0 2024-03-25T19:37:05.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.624+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3d80ab950 con 0x7fd3d81430c0 2024-03-25T19:37:05.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.624+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd3d0001b00 con 0x7fd3d81430c0 2024-03-25T19:37:05.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.624+0000 7fd3ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.624+0000 7fd3ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:05.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.624+0000 7fd3ce7fc640 10 monclient(hunting): dump: 2024-03-25T19:37:05.624 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:05.624 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:05.624 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.624 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.624 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.625 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:05.625 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:05.625 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:05.625 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:05.626 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:05.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.624+0000 7fd3ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:05.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.625+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3986355531 0 0) 0x7fd3d81d7ba0 con 0x7fd3d80a98f0 2024-03-25T19:37:05.626 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.625+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3d8138330 con 0x7fd3d80a98f0 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.625+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd3d4001cf0 con 0x7fd3d80a98f0 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.625+0000 7fd3ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.625+0000 7fd3ce7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.625+0000 7fd3ce7fc640 10 monclient(hunting): dump: 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:05.627 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:05.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.633+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3890963608 0 0) 0x7fd3bc004eb0 con 0x7fd3d813f710 2024-03-25T19:37:05.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.633+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3d81d7ba0 con 0x7fd3d813f710 2024-03-25T19:37:05.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.633+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd3c8001930 con 0x7fd3d813f710 2024-03-25T19:37:05.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.633+0000 7fd3ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.633+0000 7fd3ce7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:05.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.633+0000 7fd3ce7fc640 10 monclient(hunting): dump: 2024-03-25T19:37:05.633 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:05.633 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:05.634 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.634 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.634 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.634 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:05.634 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:05.634 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:05.634 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:05.634 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:05.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.639+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3438886695 0 0) 0x7fd3d8138330 con 0x7fd3d80a98f0 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.639+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 >> v1:172.21.15.179:6789/0 conn(0x7fd3d813f710 legacy=0x7fd3d81d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.639+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 >> v1:172.21.15.52:6789/0 conn(0x7fd3d81430c0 legacy=0x7fd3d80b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.639+0000 7fd3ce7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.639+0000 7fd3ce7fc640 1 monclient: found mon.c 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.639+0000 7fd3ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.639+0000 7fd3ce7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.639+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d80b1320 con 0x7fd3d80a98f0 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.639+0000 7fd3ce7fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.639+0000 7fd3ce7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.639+0000 7fd3ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.640+0000 7fd3dfed9640 5 monclient: authenticate success, global_id 4652 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.640+0000 7fd3dfed9640 10 monclient: _renew_subs 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.640+0000 7fd3dfed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:05.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.640+0000 7fd3dfed9640 1 -- 172.21.15.52:0/2385919022 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d80b15d0 con 0x7fd3d80a98f0 2024-03-25T19:37:05.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.640+0000 7fd3dfed9640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:05.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.640+0000 7fd3dfed9640 10 monclient: _renew_subs 2024-03-25T19:37:05.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.640+0000 7fd3dfed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:05.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.640+0000 7fd3dfed9640 1 -- 172.21.15.52:0/2385919022 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d80b09d0 con 0x7fd3d80a98f0 2024-03-25T19:37:05.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.640+0000 7fd3dfed9640 1 librados: init done 2024-03-25T19:37:05.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.640+0000 7fd3dfed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:05.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.640+0000 7fd3dfed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:05.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.641+0000 7fd3dfed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:05.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.641+0000 7fd3dfed9640 1 -- 172.21.15.52:0/2385919022 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d8138190 con 0x7fd3d80a98f0 2024-03-25T19:37:05.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.983+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3d4003990 con 0x7fd3d80a98f0 2024-03-25T19:37:05.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.983+0000 7fd3ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:05.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.983+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7fd3d4003b40 con 0x7fd3d80a98f0 2024-03-25T19:37:05.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.983+0000 7fd3ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.983+0000 7fd3ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.983+0000 7fd3ce7fc640 10 monclient: dump: 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:05.984 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.983+0000 7fd3cffff640 4 set_mon_vals no callback set 2024-03-25T19:37:05.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.984+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7fd3d4003d60 con 0x7fd3d80a98f0 2024-03-25T19:37:05.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.985+0000 7fd3dd474640 10 monclient: get_auth_request con 0x7fd3bc06e160 auth_method 0 2024-03-25T19:37:05.985 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.985+0000 7fd3ce7fc640 10 client.4652.objecter ms_handle_connect 0x7fd3bc06e160 2024-03-25T19:37:05.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.989+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7fd3d40549b0 con 0x7fd3d80a98f0 2024-03-25T19:37:05.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.989+0000 7fd3ce7fc640 10 client.4652.objecter ms_dispatch 0x7fd3d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:05.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.989+0000 7fd3ce7fc640 3 client.4652.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:05.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.989+0000 7fd3ce7fc640 3 client.4652.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:05.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.989+0000 7fd3ce7fc640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-25T19:37:05.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.989+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3d40504c0 con 0x7fd3d80a98f0 2024-03-25T19:37:05.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.989+0000 7fd3ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:05.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:05.989+0000 7fd3ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:06.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.224+0000 7fd3dfed9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.224+0000 7fd3dfed9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.224+0000 7fd3dfed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:06.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.224+0000 7fd3dfed9640 1 -- 172.21.15.52:0/2385919022 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3d80b0c20 con 0x7fd3d80a98f0 2024-03-25T19:37:06.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.267+0000 7fd3ce7fc640 1 -- 172.21.15.52:0/2385919022 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7fd3d40523f0 con 0x7fd3d80a98f0 2024-03-25T19:37:06.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.267+0000 7fd3ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:06.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.267+0000 7fd3ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:06.268 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:37:06.268 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:06.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 10 librados: watch_flush enter 2024-03-25T19:37:06.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 10 librados: watch_flush exit 2024-03-25T19:37:06.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-25T19:37:06.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 10 client.4652.objecter successfully canceled tick 2024-03-25T19:37:06.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 1 -- 172.21.15.52:0/2385919022 >> v1:172.21.15.52:6800/3967427147 conn(0x7fd3bc06e160 legacy=0x7fd3bc070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 10 monclient: shutdown 2024-03-25T19:37:06.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:06.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 1 -- 172.21.15.52:0/2385919022 >> v1:172.21.15.52:6790/0 conn(0x7fd3d80a98f0 legacy=0x7fd3d81378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:06.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 1 -- 172.21.15.52:0/2385919022 shutdown_connections 2024-03-25T19:37:06.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 1 -- 172.21.15.52:0/2385919022 >> 172.21.15.52:0/2385919022 conn(0x7fd3d80b7ea0 msgr2=0x7fd3d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:06.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 1 -- 172.21.15.52:0/2385919022 shutdown_connections 2024-03-25T19:37:06.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 1 -- 172.21.15.52:0/2385919022 wait complete. 2024-03-25T19:37:06.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:06.270+0000 7fd3dfed9640 1 librados: shutdown 2024-03-25T19:37:06.281 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:13.087 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.208+0000 7f30e7ef1640 10 monclient: get_monmap_and_config 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.208+0000 7f30e7ef1640 10 monclient: build_initial_monmap 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.208+0000 7f30e7ef1640 10 monclient: monmap: 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:37:13.210057+0000 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:37:13.210057+0000 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:37:13.209 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:13.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.209+0000 7f30e7ef1640 1 Processor -- start 2024-03-25T19:37:13.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.209+0000 7f30e7ef1640 1 -- start start 2024-03-25T19:37:13.210 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.209+0000 7f30e7ef1640 10 monclient: init 2024-03-25T19:37:13.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.210+0000 7f30e7ef1640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:13.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.210+0000 7f30e7ef1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:13.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.210+0000 7f30e7ef1640 10 monclient(hunting): picked mon.noname-b con 0x7f30e00a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:13.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.210+0000 7f30e7ef1640 10 monclient(hunting): picked mon.noname-a con 0x7f30e013f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:13.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.210+0000 7f30e7ef1640 10 monclient(hunting): picked mon.noname-c con 0x7f30e01430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:13.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.210+0000 7f30e7ef1640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e0142520 con 0x7f30e013f710 2024-03-25T19:37:13.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.210+0000 7f30e7ef1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e00b28a0 con 0x7f30e01430c0 2024-03-25T19:37:13.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.210+0000 7f30e7ef1640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e00b2a30 con 0x7f30e00a98f0 2024-03-25T19:37:13.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.210+0000 7f30e7ef1640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:13.211 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.210+0000 7f30e7ef1640 10 monclient(hunting): authenticate will time out at 6481.391496s 2024-03-25T19:37:13.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.218+0000 7f30e548c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f30e013f710 0x7f30e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:38256/0 (socket says 172.21.15.52:38256) 2024-03-25T19:37:13.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.218+0000 7f30e548c640 1 -- 172.21.15.52:0/1000345628 learned_addr learned my addr 172.21.15.52:0/1000345628 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:13.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.226+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144863596 0 0) 0x7f30e0142520 con 0x7f30e013f710 2024-03-25T19:37:13.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.226+0000 7f30e4c8b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.226+0000 7f30e4c8b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.226+0000 7f30e4c8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.226+0000 7f30e4c8b640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:37:13.227 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.226+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30cc003520 con 0x7f30e013f710 2024-03-25T19:37:13.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.241+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316728419 0 0) 0x7f30e00b28a0 con 0x7f30e01430c0 2024-03-25T19:37:13.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.241+0000 7f30e4c8b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.241+0000 7f30e4c8b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.241+0000 7f30e4c8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.241+0000 7f30e4c8b640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:37:13.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.241+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e0142520 con 0x7f30e01430c0 2024-03-25T19:37:13.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.254+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557347250 0 0) 0x7f30e00b2a30 con 0x7f30e00a98f0 2024-03-25T19:37:13.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.254+0000 7f30e4c8b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.254+0000 7f30e4c8b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.254+0000 7f30e4c8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.254+0000 7f30e4c8b640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:37:13.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.254+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e00b28a0 con 0x7f30e00a98f0 2024-03-25T19:37:13.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3880468075 0 0) 0x7f30e0142520 con 0x7f30e01430c0 2024-03-25T19:37:13.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 >> v1:172.21.15.52:6790/0 conn(0x7f30e00a98f0 legacy=0x7f30e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 >> v1:172.21.15.52:6789/0 conn(0x7f30e013f710 legacy=0x7f30e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 10 monclient: _finish_hunting 0 2024-03-25T19:37:13.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 1 monclient: found mon.noname-c 2024-03-25T19:37:13.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:13.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:37:13.258 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e00b44c0 con 0x7f30e01430c0 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 10 monclient: _finish_auth 0 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 10 monclient: _check_auth_tickets 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f30d80019f0 con 0x7f30e01430c0 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e4c8b640 10 monclient: dump: 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:13.259 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:13.260 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:13.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e7ef1640 5 monclient: authenticate success, global_id 4672 2024-03-25T19:37:13.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.257+0000 7f30e7ef1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:13.270 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.270+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30d8001b50 con 0x7f30e01430c0 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.270+0000 7f30e4c8b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.270+0000 7f30e4c8b640 1 -- 172.21.15.52:0/1000345628 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f30d8001f60 con 0x7f30e01430c0 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.270+0000 7f30e7ef1640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.270+0000 7f30e7ef1640 4 set_mon_vals no callback set 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.270+0000 7f30e4c8b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.270+0000 7f30e4c8b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.270+0000 7f30e4c8b640 10 monclient: dump: 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:13.271 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:13.272 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:13.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.271+0000 7f30e7ef1640 10 monclient: shutdown 2024-03-25T19:37:13.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.271+0000 7f30e7ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:13.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.271+0000 7f30e7ef1640 1 -- 172.21.15.52:0/1000345628 >> v1:172.21.15.179:6789/0 conn(0x7f30e01430c0 legacy=0x7f30e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.271+0000 7f30e7ef1640 1 -- 172.21.15.52:0/1000345628 shutdown_connections 2024-03-25T19:37:13.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.271+0000 7f30e7ef1640 1 -- 172.21.15.52:0/1000345628 >> 172.21.15.52:0/1000345628 conn(0x7f30e00b7ea0 msgr2=0x7f30e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:13.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.271+0000 7f30e7ef1640 1 -- 172.21.15.52:0/1000345628 shutdown_connections 2024-03-25T19:37:13.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.271+0000 7f30e7ef1640 1 -- 172.21.15.52:0/1000345628 wait complete. 2024-03-25T19:37:13.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.272+0000 7f30e7ef1640 10 monclient: build_initial_monmap 2024-03-25T19:37:13.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.272+0000 7f30e7ef1640 10 monclient: monmap: 2024-03-25T19:37:13.273 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.273+0000 7f30e7ef1640 1 librados: starting msgr at 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.273+0000 7f30e7ef1640 1 librados: starting objecter 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.273+0000 7f30e7ef1640 1 Processor -- start 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.273+0000 7f30e7ef1640 1 -- start start 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.273+0000 7f30e7ef1640 1 librados: setting wanted keys 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.273+0000 7f30e7ef1640 1 librados: calling monclient init 2024-03-25T19:37:13.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.273+0000 7f30e7ef1640 10 monclient: init 2024-03-25T19:37:13.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.273+0000 7f30e7ef1640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:13.275 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.273+0000 7f30e7ef1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:13.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.274+0000 7f30e7ef1640 10 monclient(hunting): picked mon.noname-c con 0x7f30e00a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:13.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.274+0000 7f30e7ef1640 10 monclient(hunting): picked mon.noname-b con 0x7f30e013f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:13.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.274+0000 7f30e7ef1640 10 monclient(hunting): picked mon.noname-a con 0x7f30e01430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:13.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.274+0000 7f30e7ef1640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e013c190 con 0x7f30e01430c0 2024-03-25T19:37:13.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.274+0000 7f30e7ef1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e00b7cf0 con 0x7f30e013f710 2024-03-25T19:37:13.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.274+0000 7f30e7ef1640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e013ad50 con 0x7f30e00a98f0 2024-03-25T19:37:13.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.274+0000 7f30e7ef1640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:13.276 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.274+0000 7f30e7ef1640 10 monclient(hunting): authenticate will time out at 6481.455201s 2024-03-25T19:37:13.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.277+0000 7f30e648e640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f30e01430c0 0x7f30e01e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:38282/0 (socket says 172.21.15.52:38282) 2024-03-25T19:37:13.278 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.277+0000 7f30e648e640 1 -- 172.21.15.52:0/21400241 learned_addr learned my addr 172.21.15.52:0/21400241 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:13.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.280+0000 7f30d67fc640 10 client.?.objecter ms_handle_connect 0x7f30e01430c0 2024-03-25T19:37:13.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.280+0000 7f30d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.286 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.286+0000 7f30d67fc640 10 client.?.objecter ms_handle_connect 0x7f30e013f710 2024-03-25T19:37:13.286 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.286+0000 7f30d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.291+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249775209 0 0) 0x7f30e00b44c0 con 0x7f30e01430c0 2024-03-25T19:37:13.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.291+0000 7f30d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.291+0000 7f30d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.291+0000 7f30d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.291+0000 7f30d67fc640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:37:13.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.291+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30c4004eb0 con 0x7f30e01430c0 2024-03-25T19:37:13.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.293+0000 7f30d67fc640 10 client.?.objecter ms_handle_connect 0x7f30e00a98f0 2024-03-25T19:37:13.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.293+0000 7f30d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:13.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.299+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139133358 0 0) 0x7f30e013ad50 con 0x7f30e00a98f0 2024-03-25T19:37:13.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.299+0000 7f30d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.299+0000 7f30d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.299+0000 7f30d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.299+0000 7f30d67fc640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:37:13.300 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.299+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e00b44c0 con 0x7f30e00a98f0 2024-03-25T19:37:13.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.306+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036420025 0 0) 0x7f30e00b7cf0 con 0x7f30e013f710 2024-03-25T19:37:13.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.306+0000 7f30d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:13.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.306+0000 7f30d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:13.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.306+0000 7f30d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:13.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.306+0000 7f30d67fc640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:37:13.307 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.306+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e013ad50 con 0x7f30e013f710 2024-03-25T19:37:13.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.323+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3723918910 0 0) 0x7f30e00b44c0 con 0x7f30e00a98f0 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.323+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30e00b7cf0 con 0x7f30e00a98f0 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.323+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f30dc0018f0 con 0x7f30e00a98f0 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.323+0000 7f30d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.323+0000 7f30d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.323+0000 7f30d67fc640 10 monclient(hunting): dump: 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:13.324 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.323+0000 7f30d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.329+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1563504412 0 0) 0x7f30e013ad50 con 0x7f30e013f710 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.329+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30e00b44c0 con 0x7f30e013f710 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.329+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f30d00017b0 con 0x7f30e013f710 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.329+0000 7f30d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.329+0000 7f30d67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.329+0000 7f30d67fc640 10 monclient(hunting): dump: 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.330 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:13.331 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:13.331 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:13.331 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:13.331 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:13.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.330+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3135634218 0 0) 0x7f30c4004eb0 con 0x7f30e01430c0 2024-03-25T19:37:13.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.330+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30e013ad50 con 0x7f30e01430c0 2024-03-25T19:37:13.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.330+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f30d8001d40 con 0x7f30e01430c0 2024-03-25T19:37:13.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.330+0000 7f30d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.331 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.330+0000 7f30d67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:13.332 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.330+0000 7f30d67fc640 10 monclient(hunting): dump: 2024-03-25T19:37:13.332 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:13.332 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:13.332 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.332 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.332 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.332 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:13.332 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:13.332 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:13.332 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:13.332 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2998422412 0 0) 0x7f30e00b7cf0 con 0x7f30e00a98f0 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 >> v1:172.21.15.179:6789/0 conn(0x7f30e013f710 legacy=0x7f30e01df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 >> v1:172.21.15.52:6789/0 conn(0x7f30e01430c0 legacy=0x7f30e01e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30d67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30d67fc640 1 monclient: found mon.c 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30d67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e013afc0 con 0x7f30e00a98f0 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30d67fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30d67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30e7ef1640 5 monclient: authenticate success, global_id 4658 2024-03-25T19:37:13.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30e7ef1640 10 monclient: _renew_subs 2024-03-25T19:37:13.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30e7ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:13.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30e7ef1640 1 -- 172.21.15.52:0/21400241 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e013b2a0 con 0x7f30e00a98f0 2024-03-25T19:37:13.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30e7ef1640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:13.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30e7ef1640 10 monclient: _renew_subs 2024-03-25T19:37:13.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30e7ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:13.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30e7ef1640 1 -- 172.21.15.52:0/21400241 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e012fa90 con 0x7f30e00a98f0 2024-03-25T19:37:13.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.347+0000 7f30e7ef1640 1 librados: init done 2024-03-25T19:37:13.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.348+0000 7f30e7ef1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:13.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.348+0000 7f30e7ef1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:13.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.348+0000 7f30e7ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:13.350 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.348+0000 7f30e7ef1640 1 -- 172.21.15.52:0/21400241 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e00b3ef0 con 0x7f30e00a98f0 2024-03-25T19:37:13.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.673+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30dc003610 con 0x7f30e00a98f0 2024-03-25T19:37:13.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.673+0000 7f30d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:13.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.673+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f30dc003920 con 0x7f30e00a98f0 2024-03-25T19:37:13.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.673+0000 7f30d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:13.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.673+0000 7f30d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.673+0000 7f30d67fc640 10 monclient: dump: 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:13.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.673+0000 7f30d7fff640 4 set_mon_vals no callback set 2024-03-25T19:37:13.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.674+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f30dc0198f0 con 0x7f30e00a98f0 2024-03-25T19:37:13.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.674+0000 7f30e548c640 10 monclient: get_auth_request con 0x7f30c406e0f0 auth_method 0 2024-03-25T19:37:13.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.675+0000 7f30d67fc640 10 client.4658.objecter ms_handle_connect 0x7f30c406e0f0 2024-03-25T19:37:13.681 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.681+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f30dc0535e0 con 0x7f30e00a98f0 2024-03-25T19:37:13.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.681+0000 7f30d67fc640 10 client.4658.objecter ms_dispatch 0x7f30e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:13.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.681+0000 7f30d67fc640 3 client.4658.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:13.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.681+0000 7f30d67fc640 3 client.4658.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:13.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.681+0000 7f30d67fc640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-25T19:37:13.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.681+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30dc04f0f0 con 0x7f30e00a98f0 2024-03-25T19:37:13.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.681+0000 7f30d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:13.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.681+0000 7f30d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:13.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.923+0000 7f30e7ef1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:13.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.923+0000 7f30e7ef1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:13.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.923+0000 7f30e7ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:13.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.923+0000 7f30e7ef1640 1 -- 172.21.15.52:0/21400241 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30e00627b0 con 0x7f30e00a98f0 2024-03-25T19:37:13.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.958+0000 7f30d67fc640 1 -- 172.21.15.52:0/21400241 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f30dc051020 con 0x7f30e00a98f0 2024-03-25T19:37:13.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.958+0000 7f30d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:13.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.958+0000 7f30d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:13.959 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:37:13.959 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:13.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.960+0000 7f30e7ef1640 10 librados: watch_flush enter 2024-03-25T19:37:13.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.960+0000 7f30e7ef1640 10 librados: watch_flush exit 2024-03-25T19:37:13.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.960+0000 7f30e7ef1640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-25T19:37:13.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.960+0000 7f30e7ef1640 10 client.4658.objecter successfully canceled tick 2024-03-25T19:37:13.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.961+0000 7f30e7ef1640 1 -- 172.21.15.52:0/21400241 >> v1:172.21.15.52:6800/3967427147 conn(0x7f30c406e0f0 legacy=0x7f30c40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.961+0000 7f30e7ef1640 10 monclient: shutdown 2024-03-25T19:37:13.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.961+0000 7f30e7ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:13.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.961+0000 7f30e7ef1640 1 -- 172.21.15.52:0/21400241 >> v1:172.21.15.52:6790/0 conn(0x7f30e00a98f0 legacy=0x7f30e013ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:13.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.961+0000 7f30e7ef1640 1 -- 172.21.15.52:0/21400241 shutdown_connections 2024-03-25T19:37:13.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.961+0000 7f30e7ef1640 1 -- 172.21.15.52:0/21400241 >> 172.21.15.52:0/21400241 conn(0x7f30e00b7ea0 msgr2=0x7f30e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:13.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.961+0000 7f30e7ef1640 1 -- 172.21.15.52:0/21400241 shutdown_connections 2024-03-25T19:37:13.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.961+0000 7f30e7ef1640 1 -- 172.21.15.52:0/21400241 wait complete. 2024-03-25T19:37:13.963 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:13.961+0000 7f30e7ef1640 1 librados: shutdown 2024-03-25T19:37:13.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:18.029 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:10:11.628 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.041 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:10:11.640 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.070 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:10:11.669 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.090 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:10:11.633 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.100 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:10:11.642 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.105 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:10:11.704 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.118 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:10:11.716 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.133 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:10:11.675 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:18.151 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:10:11.750 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:20.781 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:20.894 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.894+0000 7f0b442f6640 10 monclient: get_monmap_and_config 2024-03-25T19:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.894+0000 7f0b442f6640 10 monclient: build_initial_monmap 2024-03-25T19:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.894+0000 7f0b442f6640 10 monclient: monmap: 2024-03-25T19:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:37:20.895765+0000 2024-03-25T19:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:37:20.895765+0000 2024-03-25T19:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:37:20.895 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:37:20.896 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:20.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.894+0000 7f0b442f6640 1 Processor -- start 2024-03-25T19:37:20.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 1 -- start start 2024-03-25T19:37:20.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 10 monclient: init 2024-03-25T19:37:20.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:20.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:20.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b3c0a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:20.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b3c13f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:20.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b3c1430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:20.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c142520 con 0x7f0b3c1430c0 2024-03-25T19:37:20.897 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c0b28a0 con 0x7f0b3c0a98f0 2024-03-25T19:37:20.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c0b2a30 con 0x7f0b3c13f710 2024-03-25T19:37:20.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:20.898 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.895+0000 7f0b442f6640 10 monclient(hunting): authenticate will time out at 6489.076823s 2024-03-25T19:37:20.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.911+0000 7f0b41891640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f0b3c13f710 0x7f0b3c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:43814/0 (socket says 172.21.15.52:43814) 2024-03-25T19:37:20.911 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.911+0000 7f0b41891640 1 -- 172.21.15.52:0/694769627 learned_addr learned my addr 172.21.15.52:0/694769627 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:20.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.927+0000 7f0b41090640 1 -- 172.21.15.52:0/694769627 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172901569 0 0) 0x7f0b3c0b28a0 con 0x7f0b3c0a98f0 2024-03-25T19:37:20.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.927+0000 7f0b41090640 10 monclient(hunting): _negotiate 2024-03-25T19:37:20.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.927+0000 7f0b41090640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:20.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.927+0000 7f0b41090640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:20.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.927+0000 7f0b41090640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:37:20.927 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.927+0000 7f0b41090640 1 -- 172.21.15.52:0/694769627 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b28003520 con 0x7f0b3c0a98f0 2024-03-25T19:37:20.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.929+0000 7f0b41090640 1 -- 172.21.15.52:0/694769627 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473929841 0 0) 0x7f0b3c0b2a30 con 0x7f0b3c13f710 2024-03-25T19:37:20.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.929+0000 7f0b41090640 10 monclient(hunting): _negotiate 2024-03-25T19:37:20.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.929+0000 7f0b41090640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:20.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.929+0000 7f0b41090640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:20.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.929+0000 7f0b41090640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:37:20.930 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.929+0000 7f0b41090640 1 -- 172.21.15.52:0/694769627 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b3c0b28a0 con 0x7f0b3c13f710 2024-03-25T19:37:20.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 1 -- 172.21.15.52:0/694769627 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2662343007 0 0) 0x7f0b28003520 con 0x7f0b3c0a98f0 2024-03-25T19:37:20.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 1 -- 172.21.15.52:0/694769627 >> v1:172.21.15.52:6790/0 conn(0x7f0b3c13f710 legacy=0x7f0b3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:20.935 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 1 -- 172.21.15.52:0/694769627 >> v1:172.21.15.52:6789/0 conn(0x7f0b3c1430c0 legacy=0x7f0b3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 10 monclient: _finish_hunting 0 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 1 monclient: found mon.noname-c 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 1 -- 172.21.15.52:0/694769627 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c0b44c0 con 0x7f0b3c0a98f0 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 10 monclient: _finish_auth 0 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 10 monclient: _check_auth_tickets 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 1 -- 172.21.15.52:0/694769627 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0b38001690 con 0x7f0b3c0a98f0 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:20.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b41090640 10 monclient: dump: 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b442f6640 5 monclient: authenticate success, global_id 4678 2024-03-25T19:37:20.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.934+0000 7f0b442f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.967+0000 7f0b41090640 1 -- 172.21.15.52:0/694769627 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b380017f0 con 0x7f0b3c0a98f0 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.967+0000 7f0b41090640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.967+0000 7f0b41090640 1 -- 172.21.15.52:0/694769627 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0b38001c00 con 0x7f0b3c0a98f0 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.967+0000 7f0b442f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.967+0000 7f0b442f6640 4 set_mon_vals no callback set 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.967+0000 7f0b41090640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.967+0000 7f0b41090640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.967+0000 7f0b41090640 10 monclient: dump: 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:20.968 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:20.969 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:20.969 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:20.969 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:20.969 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:20.969 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:20.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.968+0000 7f0b442f6640 10 monclient: shutdown 2024-03-25T19:37:20.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.968+0000 7f0b442f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:20.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.968+0000 7f0b442f6640 1 -- 172.21.15.52:0/694769627 >> v1:172.21.15.179:6789/0 conn(0x7f0b3c0a98f0 legacy=0x7f0b3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:20.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.968+0000 7f0b442f6640 1 -- 172.21.15.52:0/694769627 shutdown_connections 2024-03-25T19:37:20.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.968+0000 7f0b442f6640 1 -- 172.21.15.52:0/694769627 >> 172.21.15.52:0/694769627 conn(0x7f0b3c0b7ea0 msgr2=0x7f0b3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:20.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.968+0000 7f0b442f6640 1 -- 172.21.15.52:0/694769627 shutdown_connections 2024-03-25T19:37:20.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.968+0000 7f0b442f6640 1 -- 172.21.15.52:0/694769627 wait complete. 2024-03-25T19:37:20.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.969+0000 7f0b442f6640 10 monclient: build_initial_monmap 2024-03-25T19:37:20.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.969+0000 7f0b442f6640 10 monclient: monmap: 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.969+0000 7f0b442f6640 1 librados: starting msgr at 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.969+0000 7f0b442f6640 1 librados: starting objecter 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.969+0000 7f0b442f6640 1 Processor -- start 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.969+0000 7f0b442f6640 1 -- start start 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.969+0000 7f0b442f6640 1 librados: setting wanted keys 2024-03-25T19:37:20.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.969+0000 7f0b442f6640 1 librados: calling monclient init 2024-03-25T19:37:20.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.969+0000 7f0b442f6640 10 monclient: init 2024-03-25T19:37:20.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.970+0000 7f0b442f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:20.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.970+0000 7f0b442f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:20.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.970+0000 7f0b442f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b3c0a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:20.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.970+0000 7f0b442f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b3c13f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:20.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.970+0000 7f0b442f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b3c1430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:20.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.970+0000 7f0b442f6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c138eb0 con 0x7f0b3c13f710 2024-03-25T19:37:20.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.970+0000 7f0b442f6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c139120 con 0x7f0b3c1430c0 2024-03-25T19:37:20.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.970+0000 7f0b442f6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3c132c50 con 0x7f0b3c0a98f0 2024-03-25T19:37:20.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.970+0000 7f0b442f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:20.973 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.970+0000 7f0b442f6640 10 monclient(hunting): authenticate will time out at 6489.151284s 2024-03-25T19:37:20.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.980+0000 7f0b42893640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f0b3c1430c0 0x7f0b3c132540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:50222/0 (socket says 172.21.15.52:50222) 2024-03-25T19:37:20.980 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.980+0000 7f0b42893640 1 -- 172.21.15.52:0/2189935361 learned_addr learned my addr 172.21.15.52:0/2189935361 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:20.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.981+0000 7f0b32ffd640 10 client.?.objecter ms_handle_connect 0x7f0b3c0a98f0 2024-03-25T19:37:20.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.981+0000 7f0b32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:20.988 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.988+0000 7f0b32ffd640 10 client.?.objecter ms_handle_connect 0x7f0b3c1430c0 2024-03-25T19:37:20.989 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.988+0000 7f0b32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:20.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.991+0000 7f0b32ffd640 10 client.?.objecter ms_handle_connect 0x7f0b3c13f710 2024-03-25T19:37:20.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.991+0000 7f0b32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:20.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.998+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485774447 0 0) 0x7f0b3c139120 con 0x7f0b3c1430c0 2024-03-25T19:37:20.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.998+0000 7f0b32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:20.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.998+0000 7f0b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:20.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.998+0000 7f0b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:20.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.998+0000 7f0b32ffd640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:37:20.999 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:20.998+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b20004eb0 con 0x7f0b3c1430c0 2024-03-25T19:37:21.001 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.001+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522396237 0 0) 0x7f0b3c132c50 con 0x7f0b3c0a98f0 2024-03-25T19:37:21.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.001+0000 7f0b32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.001+0000 7f0b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.001+0000 7f0b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.001+0000 7f0b32ffd640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:37:21.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.001+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b3c139120 con 0x7f0b3c0a98f0 2024-03-25T19:37:21.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.004+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929336383 0 0) 0x7f0b3c138eb0 con 0x7f0b3c13f710 2024-03-25T19:37:21.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.004+0000 7f0b32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:21.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.004+0000 7f0b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:21.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.004+0000 7f0b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:21.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.004+0000 7f0b32ffd640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:37:21.005 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.004+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b3c132c50 con 0x7f0b3c13f710 2024-03-25T19:37:21.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.023+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3557844046 0 0) 0x7f0b20004eb0 con 0x7f0b3c1430c0 2024-03-25T19:37:21.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.023+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b3c138eb0 con 0x7f0b3c1430c0 2024-03-25T19:37:21.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.023+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0b34001c30 con 0x7f0b3c1430c0 2024-03-25T19:37:21.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.023+0000 7f0b32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.023+0000 7f0b32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:21.024 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.023+0000 7f0b32ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:21.024 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:21.025 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:21.025 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:21.025 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:21.025 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.025 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:21.025 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:21.025 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:21.025 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:21.025 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:21.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.023+0000 7f0b32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:21.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.035+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3200349749 0 0) 0x7f0b3c139120 con 0x7f0b3c0a98f0 2024-03-25T19:37:21.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.035+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b20004eb0 con 0x7f0b3c0a98f0 2024-03-25T19:37:21.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.035+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0b38001b40 con 0x7f0b3c0a98f0 2024-03-25T19:37:21.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.035+0000 7f0b32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.035+0000 7f0b32ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:21.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.035+0000 7f0b32ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:21.036 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:21.036 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:21.036 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:21.036 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:21.036 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.036 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:21.036 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:21.036 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:21.036 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:21.036 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.037+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2613572397 0 0) 0x7f0b3c132c50 con 0x7f0b3c13f710 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.037+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b3c139120 con 0x7f0b3c13f710 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.037+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0b2c001930 con 0x7f0b3c13f710 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.037+0000 7f0b32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.037+0000 7f0b32ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.037+0000 7f0b32ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.038 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:21.039 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:21.039 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:21.039 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:21.039 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:21.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.042+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 7157757 0 0) 0x7f0b3c138eb0 con 0x7f0b3c1430c0 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.042+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 >> v1:172.21.15.52:6790/0 conn(0x7f0b3c0a98f0 legacy=0x7f0b3c137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.042+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 >> v1:172.21.15.52:6789/0 conn(0x7f0b3c13f710 legacy=0x7f0b3c131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.042+0000 7f0b32ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.042+0000 7f0b32ffd640 1 monclient: found mon.b 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.042+0000 7f0b32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.042+0000 7f0b32ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.042+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c139400 con 0x7f0b3c1430c0 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.042+0000 7f0b32ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.043+0000 7f0b32ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.043+0000 7f0b32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.043+0000 7f0b442f6640 5 monclient: authenticate success, global_id 4681 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.043+0000 7f0b442f6640 10 monclient: _renew_subs 2024-03-25T19:37:21.044 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.043+0000 7f0b442f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:21.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.043+0000 7f0b442f6640 1 -- 172.21.15.52:0/2189935361 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3c137640 con 0x7f0b3c1430c0 2024-03-25T19:37:21.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.043+0000 7f0b442f6640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:21.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.043+0000 7f0b442f6640 10 monclient: _renew_subs 2024-03-25T19:37:21.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.043+0000 7f0b442f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:21.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.043+0000 7f0b442f6640 1 -- 172.21.15.52:0/2189935361 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c1382a0 con 0x7f0b3c1430c0 2024-03-25T19:37:21.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.043+0000 7f0b442f6640 1 librados: init done 2024-03-25T19:37:21.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.044+0000 7f0b442f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.044+0000 7f0b442f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.044+0000 7f0b442f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:21.046 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.044+0000 7f0b442f6640 1 -- 172.21.15.52:0/2189935361 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c0627b0 con 0x7f0b3c1430c0 2024-03-25T19:37:21.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.320+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b34003990 con 0x7f0b3c1430c0 2024-03-25T19:37:21.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.320+0000 7f0b32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:21.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.320+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0b34003d20 con 0x7f0b3c1430c0 2024-03-25T19:37:21.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.320+0000 7f0b32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:21.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.320+0000 7f0b32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:21.321 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.320+0000 7f0b32ffd640 10 monclient: dump: 2024-03-25T19:37:21.321 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:21.322 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:21.322 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:21.322 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:21.322 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:21.322 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:21.322 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:21.322 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:21.322 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:21.322 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:21.322 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.320+0000 7f0b4088f640 4 set_mon_vals no callback set 2024-03-25T19:37:21.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.321+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f0b34003f40 con 0x7f0b3c1430c0 2024-03-25T19:37:21.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.322+0000 7f0b42092640 10 monclient: get_auth_request con 0x7f0b2006e0f0 auth_method 0 2024-03-25T19:37:21.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.322+0000 7f0b32ffd640 10 client.4681.objecter ms_handle_connect 0x7f0b2006e0f0 2024-03-25T19:37:21.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.326+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1974659948 0 0) 0x7f0b34054a30 con 0x7f0b3c1430c0 2024-03-25T19:37:21.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.326+0000 7f0b32ffd640 10 client.4681.objecter ms_dispatch 0x7f0b3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:21.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.326+0000 7f0b32ffd640 3 client.4681.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:21.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.326+0000 7f0b32ffd640 3 client.4681.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:21.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.326+0000 7f0b32ffd640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T19:37:21.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.327+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b34050540 con 0x7f0b3c1430c0 2024-03-25T19:37:21.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.327+0000 7f0b32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:21.328 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.327+0000 7f0b32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:21.571 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.571+0000 7f0b442f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:21.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.571+0000 7f0b442f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:21.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.571+0000 7f0b442f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:21.572 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.571+0000 7f0b442f6640 1 -- 172.21.15.52:0/2189935361 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b3c1377d0 con 0x7f0b3c1430c0 2024-03-25T19:37:21.621 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.621+0000 7f0b32ffd640 1 -- 172.21.15.52:0/2189935361 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1282946072) 0x7f0b34052470 con 0x7f0b3c1430c0 2024-03-25T19:37:21.621 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.621+0000 7f0b32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:21.621 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.621+0000 7f0b32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:21.622 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:37:21.622 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:27:29.208982+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:27:29.208982+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:21.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 10 librados: watch_flush enter 2024-03-25T19:37:21.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 10 librados: watch_flush exit 2024-03-25T19:37:21.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T19:37:21.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 10 client.4681.objecter successfully canceled tick 2024-03-25T19:37:21.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 1 -- 172.21.15.52:0/2189935361 >> v1:172.21.15.52:6800/3967427147 conn(0x7f0b2006e0f0 legacy=0x7f0b200705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 10 monclient: shutdown 2024-03-25T19:37:21.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:21.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 1 -- 172.21.15.52:0/2189935361 >> v1:172.21.15.179:6789/0 conn(0x7f0b3c1430c0 legacy=0x7f0b3c132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:21.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 1 -- 172.21.15.52:0/2189935361 shutdown_connections 2024-03-25T19:37:21.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 1 -- 172.21.15.52:0/2189935361 >> 172.21.15.52:0/2189935361 conn(0x7f0b3c0b7ea0 msgr2=0x7f0b3c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:21.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 1 -- 172.21.15.52:0/2189935361 shutdown_connections 2024-03-25T19:37:21.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 1 -- 172.21.15.52:0/2189935361 wait complete. 2024-03-25T19:37:21.625 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:21.623+0000 7f0b442f6640 1 librados: shutdown 2024-03-25T19:37:21.636 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:28.443 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:28.558 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.558+0000 7f068fcd0640 10 monclient: get_monmap_and_config 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.558+0000 7f068fcd0640 10 monclient: build_initial_monmap 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.558+0000 7f068fcd0640 10 monclient: monmap: 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:37:28.559865+0000 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:37:28.559865+0000 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:37:28.559 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.559+0000 7f068fcd0640 1 Processor -- start 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.559+0000 7f068fcd0640 1 -- start start 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.559+0000 7f068fcd0640 10 monclient: init 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.560+0000 7f068fcd0640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.560+0000 7f068fcd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.560+0000 7f068fcd0640 10 monclient(hunting): picked mon.noname-b con 0x7f06880a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.560+0000 7f068fcd0640 10 monclient(hunting): picked mon.noname-c con 0x7f068813f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.560+0000 7f068fcd0640 10 monclient(hunting): picked mon.noname-a con 0x7f06881430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.560+0000 7f068fcd0640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688142520 con 0x7f06881430c0 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.560+0000 7f068fcd0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06880b28a0 con 0x7f068813f710 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.560+0000 7f068fcd0640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06880b2a30 con 0x7f06880a98f0 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.560+0000 7f068fcd0640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:28.561 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.560+0000 7f068fcd0640 10 monclient(hunting): authenticate will time out at 6496.741067s 2024-03-25T19:37:28.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.570+0000 7f068da6c640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f06880a98f0 0x7f06880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:41380/0 (socket says 172.21.15.52:41380) 2024-03-25T19:37:28.570 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.570+0000 7f068da6c640 1 -- 172.21.15.52:0/1388866632 learned_addr learned my addr 172.21.15.52:0/1388866632 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:28.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.587+0000 7f068ca6a640 1 -- 172.21.15.52:0/1388866632 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020647770 0 0) 0x7f06880b28a0 con 0x7f068813f710 2024-03-25T19:37:28.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.587+0000 7f068ca6a640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.587+0000 7f068ca6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.587+0000 7f068ca6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.587+0000 7f068ca6a640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:37:28.588 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.587+0000 7f068ca6a640 1 -- 172.21.15.52:0/1388866632 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0674003520 con 0x7f068813f710 2024-03-25T19:37:28.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.597+0000 7f068ca6a640 1 -- 172.21.15.52:0/1388866632 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845878897 0 0) 0x7f06880b2a30 con 0x7f06880a98f0 2024-03-25T19:37:28.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.597+0000 7f068ca6a640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.597+0000 7f068ca6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.597+0000 7f068ca6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.597+0000 7f068ca6a640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:37:28.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.597+0000 7f068ca6a640 1 -- 172.21.15.52:0/1388866632 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06880b28a0 con 0x7f06880a98f0 2024-03-25T19:37:28.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 1 -- 172.21.15.52:0/1388866632 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 960884819 0 0) 0x7f06880b28a0 con 0x7f06880a98f0 2024-03-25T19:37:28.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 1 -- 172.21.15.52:0/1388866632 >> v1:172.21.15.179:6789/0 conn(0x7f068813f710 legacy=0x7f06881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 1 -- 172.21.15.52:0/1388866632 >> v1:172.21.15.52:6789/0 conn(0x7f06881430c0 legacy=0x7f068814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 10 monclient: _finish_hunting 0 2024-03-25T19:37:28.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 1 monclient: found mon.noname-b 2024-03-25T19:37:28.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:28.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:37:28.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 1 -- 172.21.15.52:0/1388866632 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06880b44c0 con 0x7f06880a98f0 2024-03-25T19:37:28.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 10 monclient: _finish_auth 0 2024-03-25T19:37:28.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 10 monclient: _check_auth_tickets 2024-03-25T19:37:28.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 1 -- 172.21.15.52:0/1388866632 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f06840019f0 con 0x7f06880a98f0 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068fcd0640 5 monclient: authenticate success, global_id 4667 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068fcd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068ca6a640 10 monclient: dump: 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:28.610 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.608+0000 7f068fcd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:28.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.638+0000 7f068ca6a640 1 -- 172.21.15.52:0/1388866632 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0684001b50 con 0x7f06880a98f0 2024-03-25T19:37:28.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.638+0000 7f068ca6a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.638+0000 7f068ca6a640 1 -- 172.21.15.52:0/1388866632 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0684001f20 con 0x7f06880a98f0 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.638+0000 7f068ca6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.638+0000 7f068ca6a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.638+0000 7f068ca6a640 10 monclient: dump: 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:28.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.638+0000 7f068fcd0640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:28.640 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.638+0000 7f068fcd0640 4 set_mon_vals no callback set 2024-03-25T19:37:28.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.639+0000 7f068fcd0640 10 monclient: shutdown 2024-03-25T19:37:28.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.639+0000 7f068fcd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:28.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.639+0000 7f068fcd0640 1 -- 172.21.15.52:0/1388866632 >> v1:172.21.15.52:6790/0 conn(0x7f06880a98f0 legacy=0x7f06880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.639+0000 7f068fcd0640 1 -- 172.21.15.52:0/1388866632 shutdown_connections 2024-03-25T19:37:28.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.639+0000 7f068fcd0640 1 -- 172.21.15.52:0/1388866632 >> 172.21.15.52:0/1388866632 conn(0x7f06880b7ea0 msgr2=0x7f06880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:28.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.639+0000 7f068fcd0640 1 -- 172.21.15.52:0/1388866632 shutdown_connections 2024-03-25T19:37:28.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.639+0000 7f068fcd0640 1 -- 172.21.15.52:0/1388866632 wait complete. 2024-03-25T19:37:28.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.640+0000 7f068fcd0640 10 monclient: build_initial_monmap 2024-03-25T19:37:28.641 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.640+0000 7f068fcd0640 10 monclient: monmap: 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.640+0000 7f068fcd0640 1 librados: starting msgr at 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.640+0000 7f068fcd0640 1 librados: starting objecter 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.640+0000 7f068fcd0640 1 Processor -- start 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.640+0000 7f068fcd0640 1 -- start start 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.640+0000 7f068fcd0640 1 librados: setting wanted keys 2024-03-25T19:37:28.642 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.640+0000 7f068fcd0640 1 librados: calling monclient init 2024-03-25T19:37:28.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.640+0000 7f068fcd0640 10 monclient: init 2024-03-25T19:37:28.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.641+0000 7f068fcd0640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:28.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.641+0000 7f068fcd0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:28.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.641+0000 7f068fcd0640 10 monclient(hunting): picked mon.noname-b con 0x7f06880a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:28.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.641+0000 7f068fcd0640 10 monclient(hunting): picked mon.noname-c con 0x7f068813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:28.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.641+0000 7f068fcd0640 10 monclient(hunting): picked mon.noname-a con 0x7f06881430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:28.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.641+0000 7f068fcd0640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06881d7750 con 0x7f06881430c0 2024-03-25T19:37:28.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.641+0000 7f068fcd0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0688136750 con 0x7f06880a98f0 2024-03-25T19:37:28.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.641+0000 7f068fcd0640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06881d85d0 con 0x7f068813f710 2024-03-25T19:37:28.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.641+0000 7f068fcd0640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:28.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.641+0000 7f068fcd0640 10 monclient(hunting): authenticate will time out at 6496.822721s 2024-03-25T19:37:28.649 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.649+0000 7f068da6c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f06880a98f0 0x7f06881d7ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:54918/0 (socket says 172.21.15.52:54918) 2024-03-25T19:37:28.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.649+0000 7f068da6c640 1 -- 172.21.15.52:0/3059431299 learned_addr learned my addr 172.21.15.52:0/3059431299 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:28.650 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.650+0000 7f067e7fc640 10 client.?.objecter ms_handle_connect 0x7f06880a98f0 2024-03-25T19:37:28.651 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.650+0000 7f067e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:28.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.661+0000 7f067e7fc640 10 client.?.objecter ms_handle_connect 0x7f06881430c0 2024-03-25T19:37:28.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.661+0000 7f067e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:28.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.663+0000 7f067e7fc640 10 client.?.objecter ms_handle_connect 0x7f068813f710 2024-03-25T19:37:28.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.663+0000 7f067e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:28.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.664+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396433992 0 0) 0x7f0688136750 con 0x7f06880a98f0 2024-03-25T19:37:28.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.664+0000 7f067e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.664+0000 7f067e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.664 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.664+0000 7f067e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.664+0000 7f067e7fc640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:37:28.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.664+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c004eb0 con 0x7f06880a98f0 2024-03-25T19:37:28.674 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.674+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267182227 0 0) 0x7f06881d85d0 con 0x7f068813f710 2024-03-25T19:37:28.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.674+0000 7f067e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.674+0000 7f067e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.674+0000 7f067e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.674+0000 7f067e7fc640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:37:28.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.674+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0688136750 con 0x7f068813f710 2024-03-25T19:37:28.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.675+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684479260 0 0) 0x7f06881d7750 con 0x7f06881430c0 2024-03-25T19:37:28.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.675+0000 7f067e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:28.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.675+0000 7f067e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:28.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.675+0000 7f067e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:28.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.675+0000 7f067e7fc640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:37:28.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.675+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06881d85d0 con 0x7f06881430c0 2024-03-25T19:37:28.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.685+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2478343156 0 0) 0x7f066c004eb0 con 0x7f06880a98f0 2024-03-25T19:37:28.685 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.685+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06881d7750 con 0x7f06880a98f0 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.685+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0684001f20 con 0x7f06880a98f0 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.685+0000 7f067e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.685+0000 7f067e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.685+0000 7f067e7fc640 10 monclient(hunting): dump: 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:28.686 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.685+0000 7f067e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:28.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3354199055 0 0) 0x7f06881d7750 con 0x7f06880a98f0 2024-03-25T19:37:28.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 >> v1:172.21.15.52:6790/0 conn(0x7f068813f710 legacy=0x7f0688138300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 >> v1:172.21.15.52:6789/0 conn(0x7f06881430c0 legacy=0x7f06881d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:28.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f067e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:28.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f067e7fc640 1 monclient: found mon.b 2024-03-25T19:37:28.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f067e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:28.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f067e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06880ab950 con 0x7f06880a98f0 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f067e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f067e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f067e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f068fcd0640 5 monclient: authenticate success, global_id 4687 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f068fcd0640 10 monclient: _renew_subs 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f068fcd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f068fcd0640 1 -- 172.21.15.52:0/3059431299 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06881d7a00 con 0x7f06880a98f0 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f068fcd0640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f068fcd0640 10 monclient: _renew_subs 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f068fcd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f068fcd0640 1 -- 172.21.15.52:0/3059431299 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0688138e60 con 0x7f06880a98f0 2024-03-25T19:37:28.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.697+0000 7f068fcd0640 1 librados: init done 2024-03-25T19:37:28.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.698+0000 7f068fcd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:28.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.698+0000 7f068fcd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:28.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.698+0000 7f068fcd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:28.700 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.698+0000 7f068fcd0640 1 -- 172.21.15.52:0/3059431299 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06880627b0 con 0x7f06880a98f0 2024-03-25T19:37:28.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.974+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0684004e10 con 0x7f06880a98f0 2024-03-25T19:37:28.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.974+0000 7f067e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:28.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.974+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0684005220 con 0x7f06880a98f0 2024-03-25T19:37:28.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.974+0000 7f067e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:28.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.974+0000 7f067e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:28.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.974+0000 7f067e7fc640 10 monclient: dump: 2024-03-25T19:37:28.975 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:28.976 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:28.976 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:28.976 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:28.976 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:28.976 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:28.976 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:28.976 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:28.976 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:28.976 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:28.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.974+0000 7f067ffff640 4 set_mon_vals no callback set 2024-03-25T19:37:28.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.975+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f0684005400 con 0x7f06880a98f0 2024-03-25T19:37:28.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.976+0000 7f068d26b640 10 monclient: get_auth_request con 0x7f066c06ddf0 auth_method 0 2024-03-25T19:37:28.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.976+0000 7f067e7fc640 10 client.4687.objecter ms_handle_connect 0x7f066c06ddf0 2024-03-25T19:37:28.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.981+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7f0684054d10 con 0x7f06880a98f0 2024-03-25T19:37:28.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.981+0000 7f067e7fc640 10 client.4687.objecter ms_dispatch 0x7f0688000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:37:28.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.981+0000 7f067e7fc640 3 client.4687.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:37:28.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.981+0000 7f067e7fc640 3 client.4687.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:37:28.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.981+0000 7f067e7fc640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T19:37:28.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.982+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0684050820 con 0x7f06880a98f0 2024-03-25T19:37:28.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.982+0000 7f067e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:28.982 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:28.982+0000 7f067e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:29.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.244+0000 7f068fcd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.244+0000 7f068fcd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.244+0000 7f068fcd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:29.245 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.244+0000 7f068fcd0640 1 -- 172.21.15.52:0/3059431299 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06880b7180 con 0x7f06880a98f0 2024-03-25T19:37:29.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.276+0000 7f067e7fc640 1 -- 172.21.15.52:0/3059431299 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7f0684052750 con 0x7f06880a98f0 2024-03-25T19:37:29.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.276+0000 7f067e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:29.277 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.276+0000 7f067e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:29.277 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:37:29.277 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:29.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 10 librados: watch_flush enter 2024-03-25T19:37:29.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 10 librados: watch_flush exit 2024-03-25T19:37:29.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T19:37:29.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 10 client.4687.objecter successfully canceled tick 2024-03-25T19:37:29.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 1 -- 172.21.15.52:0/3059431299 >> v1:172.21.15.52:6800/3967427147 conn(0x7f066c06ddf0 legacy=0x7f066c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 10 monclient: shutdown 2024-03-25T19:37:29.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:29.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 1 -- 172.21.15.52:0/3059431299 >> v1:172.21.15.179:6789/0 conn(0x7f06880a98f0 legacy=0x7f06881d7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:29.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 1 -- 172.21.15.52:0/3059431299 shutdown_connections 2024-03-25T19:37:29.280 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 1 -- 172.21.15.52:0/3059431299 >> 172.21.15.52:0/3059431299 conn(0x7f06880b7ea0 msgr2=0x7f0688074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:29.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 1 -- 172.21.15.52:0/3059431299 shutdown_connections 2024-03-25T19:37:29.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 1 -- 172.21.15.52:0/3059431299 wait complete. 2024-03-25T19:37:29.281 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:29.279+0000 7f068fcd0640 1 librados: shutdown 2024-03-25T19:37:29.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:36.097 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.214+0000 7f67b464e640 10 monclient: get_monmap_and_config 2024-03-25T19:37:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.214+0000 7f67b464e640 10 monclient: build_initial_monmap 2024-03-25T19:37:36.215 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.214+0000 7f67b464e640 10 monclient: monmap: 2024-03-25T19:37:36.215 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:36.215 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:36.215 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:37:36.216252+0000 2024-03-25T19:37:36.215 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:37:36.216252+0000 2024-03-25T19:37:36.215 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:36.216 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:36.216 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:36.216 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:37:36.216 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:37:36.216 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:36.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.215+0000 7f67b464e640 1 Processor -- start 2024-03-25T19:37:36.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.215+0000 7f67b464e640 1 -- start start 2024-03-25T19:37:36.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.215+0000 7f67b464e640 10 monclient: init 2024-03-25T19:37:36.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.216+0000 7f67b464e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:36.217 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.216+0000 7f67b464e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:36.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.216+0000 7f67b464e640 10 monclient(hunting): picked mon.noname-b con 0x7f67ac0b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:36.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.216+0000 7f67b464e640 10 monclient(hunting): picked mon.noname-c con 0x7f67ac0b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:36.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.216+0000 7f67b464e640 10 monclient(hunting): picked mon.noname-a con 0x7f67ac14be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:36.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.216+0000 7f67b464e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ac0af940 con 0x7f67ac14be90 2024-03-25T19:37:36.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.216+0000 7f67b464e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ac0b0a70 con 0x7f67ac0b1730 2024-03-25T19:37:36.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.216+0000 7f67b464e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ac0b0d50 con 0x7f67ac0b4d60 2024-03-25T19:37:36.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.216+0000 7f67b464e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:36.218 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.216+0000 7f67b464e640 10 monclient(hunting): authenticate will time out at 6504.397225s 2024-03-25T19:37:36.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.219+0000 7f67b2beb640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f67ac14be90 0x7f67ac14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:55928/0 (socket says 172.21.15.52:55928) 2024-03-25T19:37:36.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.219+0000 7f67b2beb640 1 -- 172.21.15.52:0/2671288577 learned_addr learned my addr 172.21.15.52:0/2671288577 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:36.241 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.241+0000 7f67b13e8640 1 -- 172.21.15.52:0/2671288577 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040626525 0 0) 0x7f67ac0b0a70 con 0x7f67ac0b1730 2024-03-25T19:37:36.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.241+0000 7f67b13e8640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.241+0000 7f67b13e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.241+0000 7f67b13e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.241+0000 7f67b13e8640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:37:36.242 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.241+0000 7f67b13e8640 1 -- 172.21.15.52:0/2671288577 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6798003520 con 0x7f67ac0b1730 2024-03-25T19:37:36.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.247+0000 7f67b13e8640 1 -- 172.21.15.52:0/2671288577 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944285796 0 0) 0x7f67ac0af940 con 0x7f67ac14be90 2024-03-25T19:37:36.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.247+0000 7f67b13e8640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.247+0000 7f67b13e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.247+0000 7f67b13e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.247+0000 7f67b13e8640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:37:36.248 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.247+0000 7f67b13e8640 1 -- 172.21.15.52:0/2671288577 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67ac0b0a70 con 0x7f67ac14be90 2024-03-25T19:37:36.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 1 -- 172.21.15.52:0/2671288577 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1218697457 0 0) 0x7f6798003520 con 0x7f67ac0b1730 2024-03-25T19:37:36.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 1 -- 172.21.15.52:0/2671288577 >> v1:172.21.15.52:6790/0 conn(0x7f67ac0b4d60 legacy=0x7f67ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 1 -- 172.21.15.52:0/2671288577 >> v1:172.21.15.52:6789/0 conn(0x7f67ac14be90 legacy=0x7f67ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 10 monclient: _finish_hunting 0 2024-03-25T19:37:36.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 1 monclient: found mon.noname-c 2024-03-25T19:37:36.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:36.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:37:36.253 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 1 -- 172.21.15.52:0/2671288577 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac0b1030 con 0x7f67ac0b1730 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 10 monclient: _finish_auth 0 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 10 monclient: _check_auth_tickets 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 1 -- 172.21.15.52:0/2671288577 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f679c001690 con 0x7f67ac0b1730 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b464e640 5 monclient: authenticate success, global_id 4690 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b464e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b13e8640 10 monclient: dump: 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:36.254 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:36.255 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:36.255 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.255 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:36.255 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:36.255 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:36.255 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:36.255 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:36.255 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.252+0000 7f67b464e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:36.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.259+0000 7f67b13e8640 1 -- 172.21.15.52:0/2671288577 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f679c0017f0 con 0x7f67ac0b1730 2024-03-25T19:37:36.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.259+0000 7f67b13e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:36.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.259+0000 7f67b13e8640 1 -- 172.21.15.52:0/2671288577 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f679c001c00 con 0x7f67ac0b1730 2024-03-25T19:37:36.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.259+0000 7f67b13e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.260+0000 7f67b13e8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:36.260 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.260+0000 7f67b13e8640 10 monclient: dump: 2024-03-25T19:37:36.261 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:36.261 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:36.261 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:36.261 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:36.261 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.261 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:36.261 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:36.261 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:36.261 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:36.261 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:36.261 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.260+0000 7f67b464e640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:36.262 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.260+0000 7f67b464e640 4 set_mon_vals no callback set 2024-03-25T19:37:36.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.260+0000 7f67b464e640 10 monclient: shutdown 2024-03-25T19:37:36.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.260+0000 7f67b464e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:36.263 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.260+0000 7f67b464e640 1 -- 172.21.15.52:0/2671288577 >> v1:172.21.15.179:6789/0 conn(0x7f67ac0b1730 legacy=0x7f67ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.261+0000 7f67b464e640 1 -- 172.21.15.52:0/2671288577 shutdown_connections 2024-03-25T19:37:36.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.261+0000 7f67b464e640 1 -- 172.21.15.52:0/2671288577 >> 172.21.15.52:0/2671288577 conn(0x7f67ac0617a0 msgr2=0x7f67ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:36.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.261+0000 7f67b464e640 1 -- 172.21.15.52:0/2671288577 shutdown_connections 2024-03-25T19:37:36.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.261+0000 7f67b464e640 1 -- 172.21.15.52:0/2671288577 wait complete. 2024-03-25T19:37:36.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.262+0000 7f67b464e640 10 monclient: build_initial_monmap 2024-03-25T19:37:36.264 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.262+0000 7f67b464e640 10 monclient: monmap: 2024-03-25T19:37:36.264 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:36.264 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:36.264 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:37:36.264 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:37:36.264 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.262+0000 7f67b464e640 1 librados: starting msgr at 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.262+0000 7f67b464e640 1 librados: starting objecter 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 1 Processor -- start 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 1 -- start start 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 1 librados: setting wanted keys 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 1 librados: calling monclient init 2024-03-25T19:37:36.265 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 10 monclient: init 2024-03-25T19:37:36.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:36.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:36.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 10 monclient(hunting): picked mon.noname-b con 0x7f67ac0b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:36.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 10 monclient(hunting): picked mon.noname-a con 0x7f67ac0b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:36.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 10 monclient(hunting): picked mon.noname-c con 0x7f67ac14be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:36.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ac13bfb0 con 0x7f67ac0b4d60 2024-03-25T19:37:36.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ac1458e0 con 0x7f67ac0b1730 2024-03-25T19:37:36.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ac12f7b0 con 0x7f67ac14be90 2024-03-25T19:37:36.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:36.267 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.263+0000 7f67b464e640 10 monclient(hunting): authenticate will time out at 6504.444665s 2024-03-25T19:37:36.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.272+0000 7f67b23ea640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f67ac0b1730 0x7f67ac13b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:35518/0 (socket says 172.21.15.52:35518) 2024-03-25T19:37:36.272 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.272+0000 7f67b23ea640 1 -- 172.21.15.52:0/3237258854 learned_addr learned my addr 172.21.15.52:0/3237258854 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:36.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.272+0000 7f67a2ffd640 10 client.?.objecter ms_handle_connect 0x7f67ac0b1730 2024-03-25T19:37:36.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.273+0000 7f67a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:36.282 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.281+0000 7f67a2ffd640 10 client.?.objecter ms_handle_connect 0x7f67ac14be90 2024-03-25T19:37:36.282 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.281+0000 7f67a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:36.286 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.286+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007213499 0 0) 0x7f67ac1458e0 con 0x7f67ac0b1730 2024-03-25T19:37:36.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.286+0000 7f67a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.286+0000 7f67a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.286+0000 7f67a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.286+0000 7f67a2ffd640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:37:36.287 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.286+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6790004eb0 con 0x7f67ac0b1730 2024-03-25T19:37:36.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.297+0000 7f67a2ffd640 10 client.?.objecter ms_handle_connect 0x7f67ac0b4d60 2024-03-25T19:37:36.297 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.297+0000 7f67a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:36.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.301+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3471190971 0 0) 0x7f6790004eb0 con 0x7f67ac0b1730 2024-03-25T19:37:36.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.301+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67ac1458e0 con 0x7f67ac0b1730 2024-03-25T19:37:36.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.301+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f67a80019d0 con 0x7f67ac0b1730 2024-03-25T19:37:36.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.301+0000 7f67a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.301+0000 7f67a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:36.302 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.301+0000 7f67a2ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:36.302 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:36.302 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:36.302 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:36.302 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:36.303 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.303 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:36.303 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:36.303 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:36.303 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:36.303 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:36.303 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.301+0000 7f67a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:36.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.304+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947409491 0 0) 0x7f67ac12f7b0 con 0x7f67ac14be90 2024-03-25T19:37:36.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.304+0000 7f67a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:36.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.304+0000 7f67a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:36.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.304+0000 7f67a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:36.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.304+0000 7f67a2ffd640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:37:36.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.305+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6790004eb0 con 0x7f67ac14be90 2024-03-25T19:37:36.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2611424752 0 0) 0x7f67ac1458e0 con 0x7f67ac0b1730 2024-03-25T19:37:36.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 >> v1:172.21.15.52:6790/0 conn(0x7f67ac14be90 legacy=0x7f67ac12f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 >> v1:172.21.15.52:6789/0 conn(0x7f67ac0b4d60 legacy=0x7f67ac12e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.310 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67a2ffd640 1 monclient: found mon.b 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67a2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac13aa90 con 0x7f67ac0b1730 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67a2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67b464e640 5 monclient: authenticate success, global_id 4693 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67b464e640 10 monclient: _renew_subs 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67b464e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67b464e640 1 -- 172.21.15.52:0/3237258854 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ac13ad70 con 0x7f67ac0b1730 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67b464e640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67b464e640 10 monclient: _renew_subs 2024-03-25T19:37:36.311 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67b464e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:36.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67b464e640 1 -- 172.21.15.52:0/3237258854 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ac145330 con 0x7f67ac0b1730 2024-03-25T19:37:36.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.309+0000 7f67b464e640 1 librados: init done 2024-03-25T19:37:36.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.310+0000 7f67b464e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:36.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.310+0000 7f67b464e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:36.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.310+0000 7f67b464e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:36.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.310+0000 7f67b464e640 1 -- 172.21.15.52:0/3237258854 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ac1401f0 con 0x7f67ac0b1730 2024-03-25T19:37:36.524 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.524+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67a8001c50 con 0x7f67ac0b1730 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.524+0000 7f67a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.524+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f67a8003a60 con 0x7f67ac0b1730 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.524+0000 7f67a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.524+0000 7f67a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.524+0000 7f67a2ffd640 10 monclient: dump: 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:36.525 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:36.526 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:36.526 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.524+0000 7f67b0be7640 4 set_mon_vals no callback set 2024-03-25T19:37:36.578 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.577+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f67a8003bc0 con 0x7f67ac0b1730 2024-03-25T19:37:36.578 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.578+0000 7f67b1be9640 10 monclient: get_auth_request con 0x7f679006dd40 auth_method 0 2024-03-25T19:37:36.579 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.578+0000 7f67a2ffd640 10 client.4693.objecter ms_handle_connect 0x7f679006dd40 2024-03-25T19:37:36.580 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.580+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7f67a8054800 con 0x7f67ac0b1730 2024-03-25T19:37:36.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.580+0000 7f67a2ffd640 10 client.4693.objecter ms_dispatch 0x7f67ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:37:36.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.580+0000 7f67a2ffd640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:37:36.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.580+0000 7f67a2ffd640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:37:36.581 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.580+0000 7f67a2ffd640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T19:37:36.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.582+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67a8050140 con 0x7f67ac0b1730 2024-03-25T19:37:36.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.582+0000 7f67a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:36.583 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.582+0000 7f67a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:36.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.831+0000 7f67b464e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:36.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.831+0000 7f67b464e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:36.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.831+0000 7f67b464e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:36.832 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.831+0000 7f67b464e640 1 -- 172.21.15.52:0/3237258854 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67ac0afbd0 con 0x7f67ac0b1730 2024-03-25T19:37:36.938 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.938+0000 7f67a2ffd640 1 -- 172.21.15.52:0/3237258854 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7f67a8052240 con 0x7f67ac0b1730 2024-03-25T19:37:36.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.938+0000 7f67a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:36.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.938+0000 7f67a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:36.939 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:37:36.939 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:36.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.940+0000 7f67b464e640 10 librados: watch_flush enter 2024-03-25T19:37:36.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.940+0000 7f67b464e640 10 librados: watch_flush exit 2024-03-25T19:37:36.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.940+0000 7f67b464e640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T19:37:36.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.940+0000 7f67b464e640 10 client.4693.objecter successfully canceled tick 2024-03-25T19:37:36.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.941+0000 7f67b464e640 1 -- 172.21.15.52:0/3237258854 >> v1:172.21.15.52:6800/3967427147 conn(0x7f679006dd40 legacy=0x7f6790070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.941+0000 7f67b464e640 10 monclient: shutdown 2024-03-25T19:37:36.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.941+0000 7f67b464e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:36.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.941+0000 7f67b464e640 1 -- 172.21.15.52:0/3237258854 >> v1:172.21.15.179:6789/0 conn(0x7f67ac0b1730 legacy=0x7f67ac13b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:36.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.941+0000 7f67b464e640 1 -- 172.21.15.52:0/3237258854 shutdown_connections 2024-03-25T19:37:36.942 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.941+0000 7f67b464e640 1 -- 172.21.15.52:0/3237258854 >> 172.21.15.52:0/3237258854 conn(0x7f67ac0617a0 msgr2=0x7f67ac13c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:36.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.941+0000 7f67b464e640 1 -- 172.21.15.52:0/3237258854 shutdown_connections 2024-03-25T19:37:36.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.941+0000 7f67b464e640 1 -- 172.21.15.52:0/3237258854 wait complete. 2024-03-25T19:37:36.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:36.941+0000 7f67b464e640 1 librados: shutdown 2024-03-25T19:37:36.953 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:38.105 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:10:31.703 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.125 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:10:31.724 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.158 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:10:31.700 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.158 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:10:31.757 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.163 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:10:31.762 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.170 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:10:31.712 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.175 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:10:31.774 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.204 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:10:31.746 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:38.215 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:10:31.814 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:43.759 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:43.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.875+0000 7f063e87d640 10 monclient: get_monmap_and_config 2024-03-25T19:37:43.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.875+0000 7f063e87d640 10 monclient: build_initial_monmap 2024-03-25T19:37:43.876 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.876+0000 7f063e87d640 10 monclient: monmap: 2024-03-25T19:37:43.877 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:43.877 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:43.877 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:37:43.877549+0000 2024-03-25T19:37:43.877 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:37:43.877549+0000 2024-03-25T19:37:43.877 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:43.877 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:43.877 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:43.877 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:37:43.877 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:37:43.877 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 1 Processor -- start 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 1 -- start start 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 10 monclient: init 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 10 monclient(hunting): picked mon.noname-a con 0x7f06380a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 10 monclient(hunting): picked mon.noname-c con 0x7f063813f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 10 monclient(hunting): picked mon.noname-b con 0x7f06381430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0638142520 con 0x7f06380a98f0 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06380b28a0 con 0x7f063813f710 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06380b2a30 con 0x7f06381430c0 2024-03-25T19:37:43.879 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:43.880 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.877+0000 7f063e87d640 10 monclient(hunting): authenticate will time out at 6512.058875s 2024-03-25T19:37:43.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.888+0000 7f06377fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f063813f710 0x7f06381429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:40036/0 (socket says 172.21.15.52:40036) 2024-03-25T19:37:43.889 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.888+0000 7f06377fe640 1 -- 172.21.15.52:0/4174093116 learned_addr learned my addr 172.21.15.52:0/4174093116 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:43.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.905+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527018486 0 0) 0x7f06380b28a0 con 0x7f063813f710 2024-03-25T19:37:43.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.905+0000 7f0636ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.905+0000 7f0636ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.905+0000 7f0636ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.905+0000 7f0636ffd640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:37:43.906 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.905+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0624003520 con 0x7f063813f710 2024-03-25T19:37:43.913 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.913+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137619737 0 0) 0x7f06380b2a30 con 0x7f06381430c0 2024-03-25T19:37:43.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.913+0000 7f0636ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.913+0000 7f0636ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.913+0000 7f0636ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.913+0000 7f0636ffd640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:37:43.914 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.913+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06380b28a0 con 0x7f06381430c0 2024-03-25T19:37:43.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.915+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711379699 0 0) 0x7f0638142520 con 0x7f06380a98f0 2024-03-25T19:37:43.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.915+0000 7f0636ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.915+0000 7f0636ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.915+0000 7f0636ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.915+0000 7f0636ffd640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:37:43.916 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.915+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06380b2a30 con 0x7f06380a98f0 2024-03-25T19:37:43.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 956297021 0 0) 0x7f0624003520 con 0x7f063813f710 2024-03-25T19:37:43.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 >> v1:172.21.15.52:6790/0 conn(0x7f06381430c0 legacy=0x7f063814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 >> v1:172.21.15.52:6789/0 conn(0x7f06380a98f0 legacy=0x7f06380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:43.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 1 monclient: found mon.noname-c 2024-03-25T19:37:43.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:43.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:37:43.923 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06380b44c0 con 0x7f063813f710 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f063e87d640 5 monclient: authenticate success, global_id 4696 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f063e87d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0628001a10 con 0x7f063813f710 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f0636ffd640 10 monclient: dump: 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:43.924 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:43.925 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:43.925 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:43.925 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:43.925 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:43.925 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.922+0000 7f063e87d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.946+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0628001b70 con 0x7f063813f710 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.946+0000 7f0636ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.946+0000 7f0636ffd640 1 -- 172.21.15.52:0/4174093116 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0628001f80 con 0x7f063813f710 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.946+0000 7f0636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.946+0000 7f0636ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.946+0000 7f0636ffd640 10 monclient: dump: 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:43.947 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:43.948 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:43.948 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:43.948 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:43.948 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:43.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.946+0000 7f063e87d640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:43.948 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.946+0000 7f063e87d640 4 set_mon_vals no callback set 2024-03-25T19:37:43.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.947+0000 7f063e87d640 10 monclient: shutdown 2024-03-25T19:37:43.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.947+0000 7f063e87d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:43.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.947+0000 7f063e87d640 1 -- 172.21.15.52:0/4174093116 >> v1:172.21.15.179:6789/0 conn(0x7f063813f710 legacy=0x7f06381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.949 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.948+0000 7f063e87d640 1 -- 172.21.15.52:0/4174093116 shutdown_connections 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.948+0000 7f063e87d640 1 -- 172.21.15.52:0/4174093116 >> 172.21.15.52:0/4174093116 conn(0x7f06380b7ea0 msgr2=0x7f06380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.948+0000 7f063e87d640 1 -- 172.21.15.52:0/4174093116 shutdown_connections 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.948+0000 7f063e87d640 1 -- 172.21.15.52:0/4174093116 wait complete. 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.948+0000 7f063e87d640 10 monclient: build_initial_monmap 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.948+0000 7f063e87d640 10 monclient: monmap: 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:37:43.950 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:37:43.951 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:43.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.949+0000 7f063e87d640 1 librados: starting msgr at 2024-03-25T19:37:43.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.949+0000 7f063e87d640 1 librados: starting objecter 2024-03-25T19:37:43.951 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.949+0000 7f063e87d640 1 Processor -- start 2024-03-25T19:37:43.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.949+0000 7f063e87d640 1 -- start start 2024-03-25T19:37:43.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.949+0000 7f063e87d640 1 librados: setting wanted keys 2024-03-25T19:37:43.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.949+0000 7f063e87d640 1 librados: calling monclient init 2024-03-25T19:37:43.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.949+0000 7f063e87d640 10 monclient: init 2024-03-25T19:37:43.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.950+0000 7f063e87d640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:43.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.950+0000 7f063e87d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:43.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.950+0000 7f063e87d640 10 monclient(hunting): picked mon.noname-b con 0x7f06380a98f0 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:43.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.950+0000 7f063e87d640 10 monclient(hunting): picked mon.noname-c con 0x7f063813f710 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:43.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.950+0000 7f063e87d640 10 monclient(hunting): picked mon.noname-a con 0x7f06381430c0 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:43.952 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.950+0000 7f063e87d640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06380abdc0 con 0x7f06381430c0 2024-03-25T19:37:43.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.950+0000 7f063e87d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06381d7730 con 0x7f06380a98f0 2024-03-25T19:37:43.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.950+0000 7f063e87d640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0638138f10 con 0x7f063813f710 2024-03-25T19:37:43.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.950+0000 7f063e87d640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:43.953 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.950+0000 7f063e87d640 10 monclient(hunting): authenticate will time out at 6512.131759s 2024-03-25T19:37:43.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.959+0000 7f0637fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f06380a98f0 0x7f0638138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:40066/0 (socket says 172.21.15.52:40066) 2024-03-25T19:37:43.959 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.959+0000 7f0637fff640 1 -- 172.21.15.52:0/3120659613 learned_addr learned my addr 172.21.15.52:0/3120659613 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:43.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.960+0000 7f0634ff9640 10 client.?.objecter ms_handle_connect 0x7f06380a98f0 2024-03-25T19:37:43.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.960+0000 7f0634ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:43.960 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.960+0000 7f0634ff9640 10 client.?.objecter ms_handle_connect 0x7f06381430c0 2024-03-25T19:37:43.961 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.960+0000 7f0634ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:43.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.966+0000 7f0634ff9640 10 client.?.objecter ms_handle_connect 0x7f063813f710 2024-03-25T19:37:43.967 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.966+0000 7f0634ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:43.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.968+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216148437 0 0) 0x7f06380abdc0 con 0x7f06381430c0 2024-03-25T19:37:43.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.968+0000 7f0634ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.968+0000 7f0634ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.968+0000 7f0634ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.968+0000 7f0634ff9640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:37:43.968 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.968+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0618004eb0 con 0x7f06381430c0 2024-03-25T19:37:43.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.975+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3679063703 0 0) 0x7f0618004eb0 con 0x7f06381430c0 2024-03-25T19:37:43.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.975+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06380abdc0 con 0x7f06381430c0 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.975+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0620001850 con 0x7f06381430c0 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.975+0000 7f0634ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.975+0000 7f0634ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.975+0000 7f0634ff9640 10 monclient(hunting): dump: 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:43.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.975+0000 7f0634ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:43.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.982+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593927874 0 0) 0x7f06381d7730 con 0x7f06380a98f0 2024-03-25T19:37:43.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.982+0000 7f0634ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.982+0000 7f0634ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.982+0000 7f0634ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.982+0000 7f0634ff9640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:37:43.983 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.982+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0618004eb0 con 0x7f06380a98f0 2024-03-25T19:37:43.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.989+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863174382 0 0) 0x7f0638138f10 con 0x7f063813f710 2024-03-25T19:37:43.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.989+0000 7f0634ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:37:43.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.989+0000 7f0634ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:43.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.989+0000 7f0634ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:43.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.989+0000 7f0634ff9640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:37:43.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.989+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06381d7730 con 0x7f063813f710 2024-03-25T19:37:43.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.990+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 992581570 0 0) 0x7f06380abdc0 con 0x7f06381430c0 2024-03-25T19:37:43.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.990+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 >> v1:172.21.15.52:6790/0 conn(0x7f063813f710 legacy=0x7f06381d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.991 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.990+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 >> v1:172.21.15.179:6789/0 conn(0x7f06380a98f0 legacy=0x7f0638138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.990+0000 7f0634ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.990+0000 7f0634ff9640 1 monclient: found mon.a 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.990+0000 7f0634ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.990+0000 7f0634ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.990+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06381390a0 con 0x7f06381430c0 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.990+0000 7f0634ff9640 10 monclient: _finish_auth 0 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.990+0000 7f0634ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.990+0000 7f0634ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.991+0000 7f063e87d640 5 monclient: authenticate success, global_id 4686 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.991+0000 7f063e87d640 10 monclient: _renew_subs 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.991+0000 7f063e87d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.991+0000 7f063e87d640 1 -- 172.21.15.52:0/3120659613 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06381d8320 con 0x7f06381430c0 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.991+0000 7f063e87d640 10 client.4686.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:43.992 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.991+0000 7f063e87d640 10 monclient: _renew_subs 2024-03-25T19:37:43.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.991+0000 7f063e87d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:37:43.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.991+0000 7f063e87d640 1 -- 172.21.15.52:0/3120659613 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06381d43c0 con 0x7f06381430c0 2024-03-25T19:37:43.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.991+0000 7f063e87d640 1 librados: init done 2024-03-25T19:37:43.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.992+0000 7f063e87d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:43.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.992+0000 7f063e87d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:43.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.992+0000 7f063e87d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:37:43.993 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:43.992+0000 7f063e87d640 1 -- 172.21.15.52:0/3120659613 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06380627b0 con 0x7f06381430c0 2024-03-25T19:37:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.303+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06200035f0 con 0x7f06381430c0 2024-03-25T19:37:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.303+0000 7f0634ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.303+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f0620003900 con 0x7f06381430c0 2024-03-25T19:37:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.303+0000 7f0634ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.303+0000 7f0634ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:44.304 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.303+0000 7f0634ff9640 10 monclient: dump: 2024-03-25T19:37:44.304 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:44.304 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:44.304 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:44.304 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:44.304 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:44.305 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:44.305 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:44.305 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:44.305 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:44.305 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:44.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.303+0000 7f06367fc640 4 set_mon_vals no callback set 2024-03-25T19:37:44.305 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.304+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f06200198d0 con 0x7f06381430c0 2024-03-25T19:37:44.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.304+0000 7f0637fff640 10 monclient: get_auth_request con 0x7f061806de80 auth_method 0 2024-03-25T19:37:44.306 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.305+0000 7f0634ff9640 10 client.4686.objecter ms_handle_connect 0x7f061806de80 2024-03-25T19:37:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.312+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7f0620053520 con 0x7f06381430c0 2024-03-25T19:37:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.312+0000 7f0634ff9640 10 client.4686.objecter ms_dispatch 0x7f0638000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:37:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.312+0000 7f0634ff9640 3 client.4686.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:37:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.312+0000 7f0634ff9640 3 client.4686.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:37:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.312+0000 7f0634ff9640 20 client.4686.objecter dump_active .. 0 homeless 2024-03-25T19:37:44.312 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.312+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f062004f030 con 0x7f06381430c0 2024-03-25T19:37:44.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.312+0000 7f0634ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:44.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.312+0000 7f0634ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:44.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.537+0000 7f063e87d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:44.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.537+0000 7f063e87d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:44.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.537+0000 7f063e87d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:37:44.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.537+0000 7f063e87d640 1 -- 172.21.15.52:0/3120659613 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06381d7930 con 0x7f06381430c0 2024-03-25T19:37:44.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.565+0000 7f0634ff9640 1 -- 172.21.15.52:0/3120659613 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7f0620050f60 con 0x7f06381430c0 2024-03-25T19:37:44.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.566+0000 7f0634ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:44.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.566+0000 7f0634ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:44.566 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:37:44.566 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 10 librados: watch_flush enter 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 10 librados: watch_flush exit 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 20 client.4686.objecter shutdown clearing up homeless session... 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 10 client.4686.objecter successfully canceled tick 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 1 -- 172.21.15.52:0/3120659613 >> v1:172.21.15.52:6800/3967427147 conn(0x7f061806de80 legacy=0x7f0618070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 10 monclient: shutdown 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 1 -- 172.21.15.52:0/3120659613 >> v1:172.21.15.52:6789/0 conn(0x7f06381430c0 legacy=0x7f06381d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 1 -- 172.21.15.52:0/3120659613 shutdown_connections 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 1 -- 172.21.15.52:0/3120659613 >> 172.21.15.52:0/3120659613 conn(0x7f06380b7ea0 msgr2=0x7f0638074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 1 -- 172.21.15.52:0/3120659613 shutdown_connections 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 1 -- 172.21.15.52:0/3120659613 wait complete. 2024-03-25T19:37:44.569 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:44.568+0000 7f063e87d640 1 librados: shutdown 2024-03-25T19:37:44.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:51.388 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:51.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.505+0000 7f171fb20640 10 monclient: get_monmap_and_config 2024-03-25T19:37:51.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.505+0000 7f171fb20640 10 monclient: build_initial_monmap 2024-03-25T19:37:51.506 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.506+0000 7f171fb20640 10 monclient: monmap: 2024-03-25T19:37:51.506 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:51.506 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:51.506 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:37:51.507549+0000 2024-03-25T19:37:51.506 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:37:51.507549+0000 2024-03-25T19:37:51.506 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:51.507 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:51.507 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:51.507 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:37:51.507 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:37:51.507 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:51.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.506+0000 7f171fb20640 1 Processor -- start 2024-03-25T19:37:51.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.506+0000 7f171fb20640 1 -- start start 2024-03-25T19:37:51.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.506+0000 7f171fb20640 10 monclient: init 2024-03-25T19:37:51.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.507+0000 7f171fb20640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:51.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.507+0000 7f171fb20640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:51.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.507+0000 7f171fb20640 10 monclient(hunting): picked mon.noname-b con 0x7f17180b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:51.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.507+0000 7f171fb20640 10 monclient(hunting): picked mon.noname-c con 0x7f17180b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:51.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.507+0000 7f171fb20640 10 monclient(hunting): picked mon.noname-a con 0x7f171814be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:51.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.507+0000 7f171fb20640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17180af940 con 0x7f171814be90 2024-03-25T19:37:51.508 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.507+0000 7f171fb20640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17180b0a70 con 0x7f17180b1730 2024-03-25T19:37:51.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.507+0000 7f171fb20640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17180b0d50 con 0x7f17180b4d60 2024-03-25T19:37:51.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.507+0000 7f171fb20640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:51.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.507+0000 7f171fb20640 10 monclient(hunting): authenticate will time out at 6519.688798s 2024-03-25T19:37:51.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.509+0000 7f171d8bc640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f17180b4d60 0x7f171813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:49458/0 (socket says 172.21.15.52:49458) 2024-03-25T19:37:51.509 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.509+0000 7f171d8bc640 1 -- 172.21.15.52:0/1928285779 learned_addr learned my addr 172.21.15.52:0/1928285779 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:51.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.520+0000 7f171c8ba640 1 -- 172.21.15.52:0/1928285779 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482584679 0 0) 0x7f17180b0d50 con 0x7f17180b4d60 2024-03-25T19:37:51.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.521+0000 7f171c8ba640 10 monclient(hunting): _negotiate 2024-03-25T19:37:51.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.521+0000 7f171c8ba640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:51.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.521+0000 7f171c8ba640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:51.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.521+0000 7f171c8ba640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:37:51.521 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.521+0000 7f171c8ba640 1 -- 172.21.15.52:0/1928285779 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1704003520 con 0x7f17180b4d60 2024-03-25T19:37:51.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.533+0000 7f171c8ba640 1 -- 172.21.15.52:0/1928285779 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824072764 0 0) 0x7f17180af940 con 0x7f171814be90 2024-03-25T19:37:51.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.533+0000 7f171c8ba640 10 monclient(hunting): _negotiate 2024-03-25T19:37:51.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.533+0000 7f171c8ba640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:51.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.533+0000 7f171c8ba640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:51.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.533+0000 7f171c8ba640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:37:51.533 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.533+0000 7f171c8ba640 1 -- 172.21.15.52:0/1928285779 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17180b0d50 con 0x7f171814be90 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 1 -- 172.21.15.52:0/1928285779 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3193922514 0 0) 0x7f1704003520 con 0x7f17180b4d60 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 1 -- 172.21.15.52:0/1928285779 >> v1:172.21.15.179:6789/0 conn(0x7f17180b1730 legacy=0x7f17180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 1 -- 172.21.15.52:0/1928285779 >> v1:172.21.15.52:6789/0 conn(0x7f171814be90 legacy=0x7f171814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 10 monclient: _finish_hunting 0 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 1 monclient: found mon.noname-b 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 1 -- 172.21.15.52:0/1928285779 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17180b1030 con 0x7f17180b4d60 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 10 monclient: _finish_auth 0 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 10 monclient: _check_auth_tickets 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 1 -- 172.21.15.52:0/1928285779 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1714001690 con 0x7f17180b4d60 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:51.544 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171c8ba640 10 monclient: dump: 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171fb20640 5 monclient: authenticate success, global_id 4685 2024-03-25T19:37:51.545 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.543+0000 7f171fb20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:51.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.549+0000 7f171c8ba640 1 -- 172.21.15.52:0/1928285779 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17140017f0 con 0x7f17180b4d60 2024-03-25T19:37:51.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.549+0000 7f171c8ba640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:51.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.549+0000 7f171c8ba640 1 -- 172.21.15.52:0/1928285779 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1714001c00 con 0x7f17180b4d60 2024-03-25T19:37:51.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.549+0000 7f171c8ba640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:51.549 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.549+0000 7f171c8ba640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.549+0000 7f171c8ba640 10 monclient: dump: 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.549+0000 7f171fb20640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:51.550 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.549+0000 7f171fb20640 4 set_mon_vals no callback set 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.550+0000 7f171fb20640 10 monclient: shutdown 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.550+0000 7f171fb20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.550+0000 7f171fb20640 1 -- 172.21.15.52:0/1928285779 >> v1:172.21.15.52:6790/0 conn(0x7f17180b4d60 legacy=0x7f171813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.550+0000 7f171fb20640 1 -- 172.21.15.52:0/1928285779 shutdown_connections 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.550+0000 7f171fb20640 1 -- 172.21.15.52:0/1928285779 >> 172.21.15.52:0/1928285779 conn(0x7f17180617a0 msgr2=0x7f1718061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.550+0000 7f171fb20640 1 -- 172.21.15.52:0/1928285779 shutdown_connections 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.550+0000 7f171fb20640 1 -- 172.21.15.52:0/1928285779 wait complete. 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.550+0000 7f171fb20640 10 monclient: build_initial_monmap 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.550+0000 7f171fb20640 10 monclient: monmap: 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:51.552 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.551+0000 7f171fb20640 1 librados: starting msgr at 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.551+0000 7f171fb20640 1 librados: starting objecter 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.551+0000 7f171fb20640 1 Processor -- start 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.551+0000 7f171fb20640 1 -- start start 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.551+0000 7f171fb20640 1 librados: setting wanted keys 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.551+0000 7f171fb20640 1 librados: calling monclient init 2024-03-25T19:37:51.553 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.551+0000 7f171fb20640 10 monclient: init 2024-03-25T19:37:51.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.552+0000 7f171fb20640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:51.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.552+0000 7f171fb20640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:51.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.552+0000 7f171fb20640 10 monclient(hunting): picked mon.noname-c con 0x7f17180b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:51.554 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.552+0000 7f171fb20640 10 monclient(hunting): picked mon.noname-a con 0x7f17180b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:51.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.552+0000 7f171fb20640 10 monclient(hunting): picked mon.noname-b con 0x7f171814be90 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:51.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.552+0000 7f171fb20640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171813ab20 con 0x7f17180b4d60 2024-03-25T19:37:51.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.552+0000 7f171fb20640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17181d7b90 con 0x7f171814be90 2024-03-25T19:37:51.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.552+0000 7f171fb20640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1718143030 con 0x7f17180b1730 2024-03-25T19:37:51.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.552+0000 7f171fb20640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:51.555 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.552+0000 7f171fb20640 10 monclient(hunting): authenticate will time out at 6519.733195s 2024-03-25T19:37:51.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.566+0000 7f171d8bc640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f17180b1730 0x7f1718139300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:49488/0 (socket says 172.21.15.52:49488) 2024-03-25T19:37:51.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.566+0000 7f171d8bc640 1 -- 172.21.15.52:0/2273282536 learned_addr learned my addr 172.21.15.52:0/2273282536 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:51.566 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.566+0000 7f170e7fc640 10 client.?.objecter ms_handle_connect 0x7f17180b1730 2024-03-25T19:37:51.567 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.566+0000 7f170e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:51.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.573+0000 7f170e7fc640 10 client.?.objecter ms_handle_connect 0x7f171814be90 2024-03-25T19:37:51.574 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.573+0000 7f170e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:51.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.576+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386779710 0 0) 0x7f1718143030 con 0x7f17180b1730 2024-03-25T19:37:51.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.577+0000 7f170e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:51.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.577+0000 7f170e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:51.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.577+0000 7f170e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:51.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.577+0000 7f170e7fc640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:37:51.577 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.577+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16fc004eb0 con 0x7f17180b1730 2024-03-25T19:37:51.578 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.578+0000 7f170e7fc640 10 client.?.objecter ms_handle_connect 0x7f17180b4d60 2024-03-25T19:37:51.578 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.578+0000 7f170e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:51.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.584+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294634375 0 0) 0x7f17181d7b90 con 0x7f171814be90 2024-03-25T19:37:51.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.584+0000 7f170e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:51.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.584+0000 7f170e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:51.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.584+0000 7f170e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:51.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.584+0000 7f170e7fc640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:37:51.584 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.584+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1718143030 con 0x7f171814be90 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.585+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2107301262 0 0) 0x7f16fc004eb0 con 0x7f17180b1730 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.585+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17181d7b90 con 0x7f17180b1730 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.585+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1714001bc0 con 0x7f17180b1730 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.585+0000 7f170e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.585+0000 7f170e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.585+0000 7f170e7fc640 10 monclient(hunting): dump: 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:51.586 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:51.587 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:51.587 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:51.587 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.585+0000 7f170e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:51.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.595+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057948970 0 0) 0x7f171813ab20 con 0x7f17180b4d60 2024-03-25T19:37:51.595 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.595+0000 7f170e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:51.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.595+0000 7f170e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:51.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.595+0000 7f170e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:51.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.595+0000 7f170e7fc640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:37:51.596 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.595+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16fc004eb0 con 0x7f17180b4d60 2024-03-25T19:37:51.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3214637498 0 0) 0x7f1718143030 con 0x7f171814be90 2024-03-25T19:37:51.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f171813ab20 con 0x7f171814be90 2024-03-25T19:37:51.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1710001b90 con 0x7f171814be90 2024-03-25T19:37:51.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:51.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:51.598 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 10 monclient(hunting): dump: 2024-03-25T19:37:51.598 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1660742163 0 0) 0x7f17181d7b90 con 0x7f17180b1730 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 >> v1:172.21.15.179:6789/0 conn(0x7f171814be90 legacy=0x7f171813b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 >> v1:172.21.15.52:6789/0 conn(0x7f17180b4d60 legacy=0x7f17181d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 1 monclient: found mon.c 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:51.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17181d87c0 con 0x7f17180b1730 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.596+0000 7f170e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.597+0000 7f171fb20640 5 monclient: authenticate success, global_id 4688 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.597+0000 7f171fb20640 10 monclient: _renew_subs 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.597+0000 7f171fb20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.597+0000 7f171fb20640 1 -- 172.21.15.52:0/2273282536 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17181432e0 con 0x7f17180b1730 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.597+0000 7f171fb20640 10 client.4688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.597+0000 7f171fb20640 10 monclient: _renew_subs 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.597+0000 7f171fb20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.597+0000 7f171fb20640 1 -- 172.21.15.52:0/2273282536 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f171813b190 con 0x7f17180b1730 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.597+0000 7f171fb20640 1 librados: init done 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.598+0000 7f171fb20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.598+0000 7f171fb20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:51.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.598+0000 7f171fb20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:51.601 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.598+0000 7f171fb20640 1 -- 172.21.15.52:0/2273282536 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17181401f0 con 0x7f17180b1730 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.628+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17140011f0 con 0x7f17180b1730 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.628+0000 7f170e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.628+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f1714004c30 con 0x7f17180b1730 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.628+0000 7f170e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.628+0000 7f170e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.628+0000 7f170e7fc640 10 monclient: dump: 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:51.629 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:51.630 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:51.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.628+0000 7f170ffff640 4 set_mon_vals no callback set 2024-03-25T19:37:51.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.629+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f171401ac80 con 0x7f17180b1730 2024-03-25T19:37:51.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.630+0000 7f171d0bb640 10 monclient: get_auth_request con 0x7f16fc06df20 auth_method 0 2024-03-25T19:37:51.630 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.630+0000 7f170e7fc640 10 client.4688.objecter ms_handle_connect 0x7f16fc06df20 2024-03-25T19:37:51.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.632+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7f1714054910 con 0x7f17180b1730 2024-03-25T19:37:51.632 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.632+0000 7f170e7fc640 10 client.4688.objecter ms_dispatch 0x7f1718000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:37:51.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.632+0000 7f170e7fc640 3 client.4688.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:37:51.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.632+0000 7f170e7fc640 3 client.4688.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:37:51.633 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.632+0000 7f170e7fc640 20 client.4688.objecter dump_active .. 0 homeless 2024-03-25T19:37:51.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.933+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1714050250 con 0x7f17180b1730 2024-03-25T19:37:51.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.933+0000 7f170e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:51.934 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:51.933+0000 7f170e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:52.176 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.176+0000 7f171fb20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.177 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.176+0000 7f171fb20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.177 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.176+0000 7f171fb20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:37:52.177 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.176+0000 7f171fb20640 1 -- 172.21.15.52:0/2273282536 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17181d7d20 con 0x7f17180b1730 2024-03-25T19:37:52.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.197+0000 7f170e7fc640 1 -- 172.21.15.52:0/2273282536 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7f1714052350 con 0x7f17180b1730 2024-03-25T19:37:52.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.197+0000 7f170e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:52.198 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.197+0000 7f170e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:52.199 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:37:52.199 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.199+0000 7f171fb20640 10 librados: watch_flush enter 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.199+0000 7f171fb20640 10 librados: watch_flush exit 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.200+0000 7f171fb20640 20 client.4688.objecter shutdown clearing up homeless session... 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.200+0000 7f171fb20640 10 client.4688.objecter successfully canceled tick 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.200+0000 7f171fb20640 1 -- 172.21.15.52:0/2273282536 >> v1:172.21.15.52:6800/3967427147 conn(0x7f16fc06df20 legacy=0x7f16fc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.200+0000 7f171fb20640 10 monclient: shutdown 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.200+0000 7f171fb20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.200+0000 7f171fb20640 1 -- 172.21.15.52:0/2273282536 >> v1:172.21.15.52:6790/0 conn(0x7f17180b1730 legacy=0x7f1718139300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.200+0000 7f171fb20640 1 -- 172.21.15.52:0/2273282536 shutdown_connections 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.200+0000 7f171fb20640 1 -- 172.21.15.52:0/2273282536 >> 172.21.15.52:0/2273282536 conn(0x7f17180617a0 msgr2=0x7f1718145330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.200+0000 7f171fb20640 1 -- 172.21.15.52:0/2273282536 shutdown_connections 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.200+0000 7f171fb20640 1 -- 172.21.15.52:0/2273282536 wait complete. 2024-03-25T19:37:52.201 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:52.200+0000 7f171fb20640 1 librados: shutdown 2024-03-25T19:37:52.213 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:58.147 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:10:51.746 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.196 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:10:51.795 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.198 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:10:51.797 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.215 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:10:51.814 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.237 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:10:51.836 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.251 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:10:51.850 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.253 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:10:51.795 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.265 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:10:51.807 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:58.290 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:10:51.833 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:59.018 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.134+0000 7f0500ed6640 10 monclient: get_monmap_and_config 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.134+0000 7f0500ed6640 10 monclient: build_initial_monmap 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.134+0000 7f0500ed6640 10 monclient: monmap: 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:37:59.135340+0000 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:37:59.135340+0000 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:37:59.134 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:37:59.135 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.135+0000 7f0500ed6640 1 Processor -- start 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.135+0000 7f0500ed6640 1 -- start start 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.135+0000 7f0500ed6640 10 monclient: init 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.136+0000 7f0500ed6640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.136+0000 7f0500ed6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.136+0000 7f0500ed6640 10 monclient(hunting): picked mon.noname-a con 0x7f04fc0b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.136+0000 7f0500ed6640 10 monclient(hunting): picked mon.noname-c con 0x7f04fc0b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.136+0000 7f0500ed6640 10 monclient(hunting): picked mon.noname-b con 0x7f04fc14be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.136+0000 7f0500ed6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc0af940 con 0x7f04fc0b4d60 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.136+0000 7f0500ed6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc0b0a70 con 0x7f04fc0b1730 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.136+0000 7f0500ed6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc0b0d50 con 0x7f04fc14be90 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.136+0000 7f0500ed6640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:59.136 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.136+0000 7f0500ed6640 10 monclient(hunting): authenticate will time out at 6527.316466s 2024-03-25T19:37:59.150 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.151+0000 7f04fa59c640 1 --1- >> v1:172.21.15.52:6789/0 conn(0x7f04fc0b4d60 0x7f04fc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6789/0 says I am v1:172.21.15.52:54544/0 (socket says 172.21.15.52:54544) 2024-03-25T19:37:59.150 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.151+0000 7f04fa59c640 1 -- 172.21.15.52:0/2331545609 learned_addr learned my addr 172.21.15.52:0/2331545609 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:59.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.162+0000 7f04f9d9b640 1 -- 172.21.15.52:0/2331545609 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402089501 0 0) 0x7f04fc0af940 con 0x7f04fc0b4d60 2024-03-25T19:37:59.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.162+0000 7f04f9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.162+0000 7f04f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.162+0000 7f04f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.162+0000 7f04f9d9b640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:37:59.162 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.162+0000 7f04f9d9b640 1 -- 172.21.15.52:0/2331545609 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e8003520 con 0x7f04fc0b4d60 2024-03-25T19:37:59.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 1 -- 172.21.15.52:0/2331545609 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3006837394 0 0) 0x7f04e8003520 con 0x7f04fc0b4d60 2024-03-25T19:37:59.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 1 -- 172.21.15.52:0/2331545609 >> v1:172.21.15.52:6790/0 conn(0x7f04fc14be90 legacy=0x7f04fc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.171 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 1 -- 172.21.15.52:0/2331545609 >> v1:172.21.15.179:6789/0 conn(0x7f04fc0b1730 legacy=0x7f04fc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 1 monclient: found mon.noname-a 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 1 -- 172.21.15.52:0/2331545609 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc0b1030 con 0x7f04fc0b4d60 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f04f9d9b640 1 -- 172.21.15.52:0/2331545609 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f04ec001690 con 0x7f04fc0b4d60 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.171+0000 7f0500ed6640 5 monclient: authenticate success, global_id 4695 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.172+0000 7f0500ed6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.172+0000 7f04f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.172+0000 7f04f9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.172+0000 7f04f9d9b640 10 monclient: dump: 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:59.172 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:59.173 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:59.173 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.173 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:59.173 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:59.173 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:59.173 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:59.173 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:59.173 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.172+0000 7f0500ed6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:59.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f04f9d9b640 1 -- 172.21.15.52:0/2331545609 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04ec0017f0 con 0x7f04fc0b4d60 2024-03-25T19:37:59.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f04f9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:59.181 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f04f9d9b640 1 -- 172.21.15.52:0/2331545609 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f04ec001c40 con 0x7f04fc0b4d60 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f04f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f04f9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f04f9d9b640 10 monclient: dump: 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f0500ed6640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:59.182 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f0500ed6640 4 set_mon_vals no callback set 2024-03-25T19:37:59.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f0500ed6640 10 monclient: shutdown 2024-03-25T19:37:59.184 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f0500ed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f0500ed6640 1 -- 172.21.15.52:0/2331545609 >> v1:172.21.15.52:6789/0 conn(0x7f04fc0b4d60 legacy=0x7f04fc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f0500ed6640 1 -- 172.21.15.52:0/2331545609 shutdown_connections 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.182+0000 7f0500ed6640 1 -- 172.21.15.52:0/2331545609 >> 172.21.15.52:0/2331545609 conn(0x7f04fc0617a0 msgr2=0x7f04fc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.183+0000 7f0500ed6640 1 -- 172.21.15.52:0/2331545609 shutdown_connections 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.183+0000 7f0500ed6640 1 -- 172.21.15.52:0/2331545609 wait complete. 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.183+0000 7f0500ed6640 10 monclient: build_initial_monmap 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.183+0000 7f0500ed6640 10 monclient: monmap: 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:37:59.185 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 1 librados: starting msgr at 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 1 librados: starting objecter 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 1 Processor -- start 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 1 -- start start 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 1 librados: setting wanted keys 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 1 librados: calling monclient init 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 10 monclient: init 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 10 monclient(hunting): picked mon.noname-a con 0x7f04fc0b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 10 monclient(hunting): picked mon.noname-b con 0x7f04fc0b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:37:59.186 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 10 monclient(hunting): picked mon.noname-c con 0x7f04fc14be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:37:59.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc13a940 con 0x7f04fc0b1730 2024-03-25T19:37:59.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc13ab90 con 0x7f04fc0b4d60 2024-03-25T19:37:59.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04fc1d7840 con 0x7f04fc14be90 2024-03-25T19:37:59.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:59.187 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.184+0000 7f0500ed6640 10 monclient(hunting): authenticate will time out at 6527.364843s 2024-03-25T19:37:59.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.193+0000 7f04f3fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f04fc0b4d60 0x7f04fc1d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:33834/0 (socket says 172.21.15.52:33834) 2024-03-25T19:37:59.192 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.193+0000 7f04f3fff640 1 -- 172.21.15.52:0/1121814222 learned_addr learned my addr 172.21.15.52:0/1121814222 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:37:59.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.196+0000 7f04f2ffd640 10 client.?.objecter ms_handle_connect 0x7f04fc14be90 2024-03-25T19:37:59.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.196+0000 7f04f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.196+0000 7f04f2ffd640 10 client.?.objecter ms_handle_connect 0x7f04fc0b4d60 2024-03-25T19:37:59.195 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.196+0000 7f04f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.197+0000 7f04f2ffd640 10 client.?.objecter ms_handle_connect 0x7f04fc0b1730 2024-03-25T19:37:59.196 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.197+0000 7f04f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:59.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.204+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000741029 0 0) 0x7f04fc1d7840 con 0x7f04fc14be90 2024-03-25T19:37:59.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.204+0000 7f04f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.203 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.204+0000 7f04f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.204+0000 7f04f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.204+0000 7f04f2ffd640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:37:59.204 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.204+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04dc004eb0 con 0x7f04fc14be90 2024-03-25T19:37:59.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.206+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555632833 0 0) 0x7f04fc13ab90 con 0x7f04fc0b4d60 2024-03-25T19:37:59.205 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.206+0000 7f04f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.206+0000 7f04f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.206+0000 7f04f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.206+0000 7f04f2ffd640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:37:59.206 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.206+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc1d7840 con 0x7f04fc0b4d60 2024-03-25T19:37:59.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.216+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369302021 0 0) 0x7f04fc13a940 con 0x7f04fc0b1730 2024-03-25T19:37:59.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.216+0000 7f04f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:59.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.216+0000 7f04f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:59.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.216+0000 7f04f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:59.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.216+0000 7f04f2ffd640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:37:59.216 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.216+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc13ab90 con 0x7f04fc0b1730 2024-03-25T19:37:59.219 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.220+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3539608995 0 0) 0x7f04fc1d7840 con 0x7f04fc0b4d60 2024-03-25T19:37:59.220 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.220+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04fc13a940 con 0x7f04fc0b4d60 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.223+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f04e40019f0 con 0x7f04fc0b4d60 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.223+0000 7f04f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.223+0000 7f04f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.223+0000 7f04f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:59.223 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:59.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.223+0000 7f04f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:59.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.225+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3619406885 0 0) 0x7f04dc004eb0 con 0x7f04fc14be90 2024-03-25T19:37:59.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.225+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04fc1d7840 con 0x7f04fc14be90 2024-03-25T19:37:59.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.225+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f04f4001b90 con 0x7f04fc14be90 2024-03-25T19:37:59.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.225+0000 7f04f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.224 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.225+0000 7f04f2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:59.225 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.225+0000 7f04f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:59.225 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:59.225 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:59.225 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:59.225 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:59.225 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.225 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:59.225 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:59.225 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:59.225 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:59.225 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:59.230 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 396193064 0 0) 0x7f04fc13a940 con 0x7f04fc0b4d60 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 >> v1:172.21.15.52:6790/0 conn(0x7f04fc14be90 legacy=0x7f04fc1e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 >> v1:172.21.15.52:6789/0 conn(0x7f04fc0b1730 legacy=0x7f04fc13bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f04f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f04f2ffd640 1 monclient: found mon.b 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f04f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f04f2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc13c240 con 0x7f04fc0b4d60 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f04f2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f04f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f04f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f0500ed6640 5 monclient: authenticate success, global_id 4711 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f0500ed6640 10 monclient: _renew_subs 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f0500ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f0500ed6640 1 -- 172.21.15.52:0/1121814222 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04fc139ce0 con 0x7f04fc0b4d60 2024-03-25T19:37:59.231 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f0500ed6640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:59.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f0500ed6640 10 monclient: _renew_subs 2024-03-25T19:37:59.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f0500ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:59.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f0500ed6640 1 -- 172.21.15.52:0/1121814222 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04fc1d8650 con 0x7f04fc0b4d60 2024-03-25T19:37:59.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.231+0000 7f0500ed6640 1 librados: init done 2024-03-25T19:37:59.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.232+0000 7f0500ed6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:59.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.232+0000 7f0500ed6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:59.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.232+0000 7f0500ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:59.232 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.232+0000 7f0500ed6640 1 -- 172.21.15.52:0/1121814222 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04fc1401f0 con 0x7f04fc0b4d60 2024-03-25T19:37:59.529 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.530+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04e40035f0 con 0x7f04fc0b4d60 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.530+0000 7f04f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.530+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f04e40039c0 con 0x7f04fc0b4d60 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.530+0000 7f04f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.530+0000 7f04f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.530+0000 7f04f2ffd640 10 monclient: dump: 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:37:59.530 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:37:59.531 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:37:59.531 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:37:59.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.530+0000 7f04f959a640 4 set_mon_vals no callback set 2024-03-25T19:37:59.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.531+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f04e4003be0 con 0x7f04fc0b4d60 2024-03-25T19:37:59.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.531+0000 7f04fa59c640 10 monclient: get_auth_request con 0x7f04dc06df60 auth_method 0 2024-03-25T19:37:59.531 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.531+0000 7f04f2ffd640 10 client.4711.objecter ms_handle_connect 0x7f04dc06df60 2024-03-25T19:37:59.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.538+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7f04e4054750 con 0x7f04fc0b4d60 2024-03-25T19:37:59.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.538+0000 7f04f2ffd640 10 client.4711.objecter ms_dispatch 0x7f04fc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:37:59.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.538+0000 7f04f2ffd640 3 client.4711.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:37:59.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.538+0000 7f04f2ffd640 3 client.4711.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:37:59.537 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.538+0000 7f04f2ffd640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T19:37:59.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.538+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04e4050260 con 0x7f04fc0b4d60 2024-03-25T19:37:59.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.538+0000 7f04f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:59.538 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.538+0000 7f04f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.774+0000 7f0500ed6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.774+0000 7f0500ed6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.774+0000 7f0500ed6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:37:59.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.774+0000 7f0500ed6640 1 -- 172.21.15.52:0/1121814222 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04fc0627b0 con 0x7f04fc0b4d60 2024-03-25T19:37:59.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.810+0000 7f04f2ffd640 1 -- 172.21.15.52:0/1121814222 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7f04e4052190 con 0x7f04fc0b4d60 2024-03-25T19:37:59.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.810+0000 7f04f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:59.809 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.810+0000 7f04f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:59.810 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:37:59.810 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 10 librados: watch_flush enter 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 10 librados: watch_flush exit 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 10 client.4711.objecter successfully canceled tick 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 1 -- 172.21.15.52:0/1121814222 >> v1:172.21.15.52:6800/3967427147 conn(0x7f04dc06df60 legacy=0x7f04dc070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 10 monclient: shutdown 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 1 -- 172.21.15.52:0/1121814222 >> v1:172.21.15.179:6789/0 conn(0x7f04fc0b4d60 legacy=0x7f04fc1d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 1 -- 172.21.15.52:0/1121814222 shutdown_connections 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 1 -- 172.21.15.52:0/1121814222 >> 172.21.15.52:0/1121814222 conn(0x7f04fc0617a0 msgr2=0x7f04fc137510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 1 -- 172.21.15.52:0/1121814222 shutdown_connections 2024-03-25T19:37:59.813 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 1 -- 172.21.15.52:0/1121814222 wait complete. 2024-03-25T19:37:59.814 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:37:59.813+0000 7f0500ed6640 1 librados: shutdown 2024-03-25T19:37:59.824 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:06.630 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:06.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.749+0000 7f6440c25640 10 monclient: get_monmap_and_config 2024-03-25T19:38:06.748 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.749+0000 7f6440c25640 10 monclient: build_initial_monmap 2024-03-25T19:38:06.749 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.749+0000 7f6440c25640 10 monclient: monmap: 2024-03-25T19:38:06.749 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:06.749 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:06.749 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:38:06.749881+0000 2024-03-25T19:38:06.749 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:38:06.749881+0000 2024-03-25T19:38:06.749 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:06.749 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:06.749 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:06.749 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:38:06.749 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:38:06.749 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:06.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.750+0000 7f6440c25640 1 Processor -- start 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.750+0000 7f6440c25640 1 -- start start 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.750+0000 7f6440c25640 10 monclient: init 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.751+0000 7f6440c25640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.751+0000 7f6440c25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.751+0000 7f6440c25640 10 monclient(hunting): picked mon.noname-a con 0x7f643c0b4d60 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.751+0000 7f6440c25640 10 monclient(hunting): picked mon.noname-c con 0x7f643c0b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.751+0000 7f6440c25640 10 monclient(hunting): picked mon.noname-b con 0x7f643c14be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.751+0000 7f6440c25640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c0af940 con 0x7f643c0b4d60 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.751+0000 7f6440c25640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c0b0a70 con 0x7f643c0b1730 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.751+0000 7f6440c25640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c0b0d50 con 0x7f643c14be90 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.751+0000 7f6440c25640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:06.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.751+0000 7f6440c25640 10 monclient(hunting): authenticate will time out at 6534.931080s 2024-03-25T19:38:06.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.761+0000 7f643ad9d640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f643c14be90 0x7f643c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:38300/0 (socket says 172.21.15.52:38300) 2024-03-25T19:38:06.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.761+0000 7f643ad9d640 1 -- 172.21.15.52:0/312138105 learned_addr learned my addr 172.21.15.52:0/312138105 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:06.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.773+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975150067 0 0) 0x7f643c0af940 con 0x7f643c0b4d60 2024-03-25T19:38:06.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.773+0000 7f6439d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.773+0000 7f6439d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.773+0000 7f6439d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.773+0000 7f6439d9b640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:38:06.773 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.773+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6428003520 con 0x7f643c0b4d60 2024-03-25T19:38:06.774 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.775+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010169746 0 0) 0x7f643c0b0d50 con 0x7f643c14be90 2024-03-25T19:38:06.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.775+0000 7f6439d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.775+0000 7f6439d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.775+0000 7f6439d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.775+0000 7f6439d9b640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:38:06.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.775+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f643c0af940 con 0x7f643c14be90 2024-03-25T19:38:06.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.776+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784712741 0 0) 0x7f643c0b0a70 con 0x7f643c0b1730 2024-03-25T19:38:06.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.776+0000 7f6439d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.775 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.776+0000 7f6439d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.776+0000 7f6439d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.776+0000 7f6439d9b640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:38:06.776 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.776+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f643c0b0d50 con 0x7f643c0b1730 2024-03-25T19:38:06.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1402470124 0 0) 0x7f643c0af940 con 0x7f643c14be90 2024-03-25T19:38:06.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 >> v1:172.21.15.179:6789/0 conn(0x7f643c0b1730 legacy=0x7f643c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 >> v1:172.21.15.52:6789/0 conn(0x7f643c0b4d60 legacy=0x7f643c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:38:06.785 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 1 monclient: found mon.noname-b 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c0b1030 con 0x7f643c14be90 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 10 monclient: _finish_auth 0 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f64340019f0 con 0x7f643c14be90 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6440c25640 5 monclient: authenticate success, global_id 4697 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6440c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6439d9b640 10 monclient: dump: 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:06.786 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:06.787 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:06.787 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:06.787 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:06.787 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:06.787 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:06.787 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:06.787 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:06.787 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.785+0000 7f6440c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:06.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6434001b50 con 0x7f643c14be90 2024-03-25T19:38:06.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6439d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:06.815 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6439d9b640 1 -- 172.21.15.52:0/312138105 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6434001f60 con 0x7f643c14be90 2024-03-25T19:38:06.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6439d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:06.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6439d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:06.816 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6439d9b640 10 monclient: dump: 2024-03-25T19:38:06.816 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:06.816 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:06.816 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:06.816 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:06.816 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:06.816 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:06.816 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:06.816 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:06.817 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:06.817 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:06.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6440c25640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:06.817 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6440c25640 4 set_mon_vals no callback set 2024-03-25T19:38:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6440c25640 10 monclient: shutdown 2024-03-25T19:38:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6440c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6440c25640 1 -- 172.21.15.52:0/312138105 >> v1:172.21.15.52:6790/0 conn(0x7f643c14be90 legacy=0x7f643c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6440c25640 1 -- 172.21.15.52:0/312138105 shutdown_connections 2024-03-25T19:38:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6440c25640 1 -- 172.21.15.52:0/312138105 >> 172.21.15.52:0/312138105 conn(0x7f643c0617a0 msgr2=0x7f643c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:06.819 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6440c25640 1 -- 172.21.15.52:0/312138105 shutdown_connections 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.816+0000 7f6440c25640 1 -- 172.21.15.52:0/312138105 wait complete. 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.817+0000 7f6440c25640 10 monclient: build_initial_monmap 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.817+0000 7f6440c25640 10 monclient: monmap: 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 1 librados: starting msgr at 2024-03-25T19:38:06.820 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 1 librados: starting objecter 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 1 Processor -- start 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 1 -- start start 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 1 librados: setting wanted keys 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 1 librados: calling monclient init 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 10 monclient: init 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 10 monclient(hunting): picked mon.noname-c con 0x7f643c0b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 10 monclient(hunting): picked mon.noname-b con 0x7f643c0b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 10 monclient(hunting): picked mon.noname-a con 0x7f643c14be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c1382d0 con 0x7f643c14be90 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c1387f0 con 0x7f643c0b4d60 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c13bba0 con 0x7f643c0b1730 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:06.821 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.818+0000 7f6440c25640 10 monclient(hunting): authenticate will time out at 6534.998695s 2024-03-25T19:38:06.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.823+0000 7f643a59c640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f643c0b1730 0x7f643c131e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:38350/0 (socket says 172.21.15.52:38350) 2024-03-25T19:38:06.822 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.823+0000 7f643a59c640 1 -- 172.21.15.52:0/1379257434 learned_addr learned my addr 172.21.15.52:0/1379257434 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:06.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.824+0000 7f6432ffd640 10 client.?.objecter ms_handle_connect 0x7f643c0b1730 2024-03-25T19:38:06.823 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.824+0000 7f6432ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:06.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.829+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672856889 0 0) 0x7f643c13bba0 con 0x7f643c0b1730 2024-03-25T19:38:06.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.829+0000 7f6432ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.829+0000 7f6432ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.829+0000 7f6432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.829+0000 7f6432ffd640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:38:06.829 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.829+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f641c004eb0 con 0x7f643c0b1730 2024-03-25T19:38:06.830 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.831+0000 7f6432ffd640 10 client.?.objecter ms_handle_connect 0x7f643c0b4d60 2024-03-25T19:38:06.831 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.831+0000 7f6432ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:06.833 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.834+0000 7f6432ffd640 10 client.?.objecter ms_handle_connect 0x7f643c14be90 2024-03-25T19:38:06.834 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.834+0000 7f6432ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:06.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.839+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 471149970 0 0) 0x7f641c004eb0 con 0x7f643c0b1730 2024-03-25T19:38:06.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.839+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f643c13bba0 con 0x7f643c0b1730 2024-03-25T19:38:06.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.839+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f642c001990 con 0x7f643c0b1730 2024-03-25T19:38:06.839 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.839+0000 7f6432ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.839+0000 7f6432ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.839+0000 7f6432ffd640 10 monclient(hunting): dump: 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.839+0000 7f6432ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709655164 0 0) 0x7f643c1387f0 con 0x7f643c0b4d60 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.840 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:38:06.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f641c004eb0 con 0x7f643c0b4d60 2024-03-25T19:38:06.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189583259 0 0) 0x7f643c1382d0 con 0x7f643c14be90 2024-03-25T19:38:06.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:06.841 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:06.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:06.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:38:06.842 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.840+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f643c1387f0 con 0x7f643c14be90 2024-03-25T19:38:06.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1805613732 0 0) 0x7f643c13bba0 con 0x7f643c0b1730 2024-03-25T19:38:06.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 >> v1:172.21.15.179:6789/0 conn(0x7f643c0b4d60 legacy=0x7f643c1d7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.844 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 >> v1:172.21.15.52:6789/0 conn(0x7f643c14be90 legacy=0x7f643c1e2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6432ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6432ffd640 1 monclient: found mon.c 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6432ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6432ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c132590 con 0x7f643c0b1730 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6432ffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6432ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6432ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6440c25640 5 monclient: authenticate success, global_id 4700 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6440c25640 10 monclient: _renew_subs 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6440c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6440c25640 1 -- 172.21.15.52:0/1379257434 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643c1327e0 con 0x7f643c0b1730 2024-03-25T19:38:06.845 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6440c25640 10 client.4700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:06.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6440c25640 10 monclient: _renew_subs 2024-03-25T19:38:06.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6440c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:38:06.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6440c25640 1 -- 172.21.15.52:0/1379257434 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f643c1d7e10 con 0x7f643c0b1730 2024-03-25T19:38:06.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.844+0000 7f6440c25640 1 librados: init done 2024-03-25T19:38:06.846 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.845+0000 7f6440c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:06.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.845+0000 7f6440c25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:06.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.845+0000 7f6440c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:38:06.847 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:06.845+0000 7f6440c25640 1 -- 172.21.15.52:0/1379257434 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c138130 con 0x7f643c0b1730 2024-03-25T19:38:07.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.156+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f642c001c50 con 0x7f643c0b1730 2024-03-25T19:38:07.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.156+0000 7f6432ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:07.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.156+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f642c0038c0 con 0x7f643c0b1730 2024-03-25T19:38:07.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.156+0000 7f6432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:07.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.156+0000 7f6432ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:07.156 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.156+0000 7f6432ffd640 10 monclient: dump: 2024-03-25T19:38:07.156 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:07.156 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:07.156 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:07.157 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:07.157 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:07.157 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:07.157 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:07.157 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:07.157 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:07.157 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:07.157 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.156+0000 7f643959a640 4 set_mon_vals no callback set 2024-03-25T19:38:07.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.157+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f642c003ae0 con 0x7f643c0b1730 2024-03-25T19:38:07.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.158+0000 7f6433fff640 10 monclient: get_auth_request con 0x7f641c06de10 auth_method 0 2024-03-25T19:38:07.158 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.158+0000 7f6432ffd640 10 client.4700.objecter ms_handle_connect 0x7f641c06de10 2024-03-25T19:38:07.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.165+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7f642c054670 con 0x7f643c0b1730 2024-03-25T19:38:07.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.165+0000 7f6432ffd640 10 client.4700.objecter ms_dispatch 0x7f643c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:38:07.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.165+0000 7f6432ffd640 3 client.4700.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:38:07.164 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.165+0000 7f6432ffd640 3 client.4700.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:38:07.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.165+0000 7f6432ffd640 20 client.4700.objecter dump_active .. 0 homeless 2024-03-25T19:38:07.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.165+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f642c0b18b0 con 0x7f643c0b1730 2024-03-25T19:38:07.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.165+0000 7f6432ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:07.165 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.165+0000 7f6432ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:07.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.396+0000 7f6440c25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:07.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.396+0000 7f6440c25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:07.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.396+0000 7f6440c25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:38:07.395 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.396+0000 7f6440c25640 1 -- 172.21.15.52:0/1379257434 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f643c137a00 con 0x7f643c0b1730 2024-03-25T19:38:07.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.435+0000 7f6432ffd640 1 -- 172.21.15.52:0/1379257434 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7f642c050180 con 0x7f643c0b1730 2024-03-25T19:38:07.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.435+0000 7f6432ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:07.434 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.435+0000 7f6432ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:07.435 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:38:07.435 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:07.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.437+0000 7f6440c25640 10 librados: watch_flush enter 2024-03-25T19:38:07.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.437+0000 7f6440c25640 10 librados: watch_flush exit 2024-03-25T19:38:07.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.437+0000 7f6440c25640 20 client.4700.objecter shutdown clearing up homeless session... 2024-03-25T19:38:07.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.437+0000 7f6440c25640 10 client.4700.objecter successfully canceled tick 2024-03-25T19:38:07.437 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.437+0000 7f6440c25640 1 -- 172.21.15.52:0/1379257434 >> v1:172.21.15.52:6800/3967427147 conn(0x7f641c06de10 legacy=0x7f641c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.437+0000 7f6440c25640 10 monclient: shutdown 2024-03-25T19:38:07.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.437+0000 7f6440c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:07.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.437+0000 7f6440c25640 1 -- 172.21.15.52:0/1379257434 >> v1:172.21.15.52:6790/0 conn(0x7f643c0b1730 legacy=0x7f643c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:07.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.437+0000 7f6440c25640 1 -- 172.21.15.52:0/1379257434 shutdown_connections 2024-03-25T19:38:07.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.437+0000 7f6440c25640 1 -- 172.21.15.52:0/1379257434 >> 172.21.15.52:0/1379257434 conn(0x7f643c0617a0 msgr2=0x7f643c0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:07.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.437+0000 7f6440c25640 1 -- 172.21.15.52:0/1379257434 shutdown_connections 2024-03-25T19:38:07.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.438+0000 7f6440c25640 1 -- 172.21.15.52:0/1379257434 wait complete. 2024-03-25T19:38:07.438 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:07.438+0000 7f6440c25640 1 librados: shutdown 2024-03-25T19:38:07.448 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:14.255 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:14.369 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.370+0000 7ff6f2038640 10 monclient: get_monmap_and_config 2024-03-25T19:38:14.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.370+0000 7ff6f2038640 10 monclient: build_initial_monmap 2024-03-25T19:38:14.370 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.370+0000 7ff6f2038640 10 monclient: monmap: 2024-03-25T19:38:14.370 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:14.370 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:14.370 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:38:14.370837+0000 2024-03-25T19:38:14.370 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:38:14.370837+0000 2024-03-25T19:38:14.370 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:14.370 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:14.371 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:14.371 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:38:14.371 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:38:14.371 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:14.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.371+0000 7ff6f2038640 1 Processor -- start 2024-03-25T19:38:14.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.371+0000 7ff6f2038640 1 -- start start 2024-03-25T19:38:14.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.371+0000 7ff6f2038640 10 monclient: init 2024-03-25T19:38:14.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.372+0000 7ff6f2038640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:14.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.372+0000 7ff6f2038640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:14.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.372+0000 7ff6f2038640 10 monclient(hunting): picked mon.noname-a con 0x7ff6ec0a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:14.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.372+0000 7ff6f2038640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ec13f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:14.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.372+0000 7ff6f2038640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ec1430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:14.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.372+0000 7ff6f2038640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec142520 con 0x7ff6ec0a98f0 2024-03-25T19:38:14.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.372+0000 7ff6f2038640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec0b28a0 con 0x7ff6ec13f710 2024-03-25T19:38:14.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.372+0000 7ff6f2038640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec0b2a30 con 0x7ff6ec1430c0 2024-03-25T19:38:14.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.372+0000 7ff6f2038640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:14.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.372+0000 7ff6f2038640 10 monclient(hunting): authenticate will time out at 6542.552293s 2024-03-25T19:38:14.376 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.377+0000 7ff6ebfff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7ff6ec1430c0 0x7ff6ec14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:42990/0 (socket says 172.21.15.52:42990) 2024-03-25T19:38:14.377 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.377+0000 7ff6ebfff640 1 -- 172.21.15.52:0/2818093067 learned_addr learned my addr 172.21.15.52:0/2818093067 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:14.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.384+0000 7ff6ea7fc640 1 -- 172.21.15.52:0/2818093067 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848381368 0 0) 0x7ff6ec0b2a30 con 0x7ff6ec1430c0 2024-03-25T19:38:14.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.384+0000 7ff6ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:14.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.384+0000 7ff6ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:14.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.384+0000 7ff6ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:14.383 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.384+0000 7ff6ea7fc640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:38:14.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.384+0000 7ff6ea7fc640 1 -- 172.21.15.52:0/2818093067 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d4003520 con 0x7ff6ec1430c0 2024-03-25T19:38:14.392 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.393+0000 7ff6ea7fc640 1 -- 172.21.15.52:0/2818093067 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132909021 0 0) 0x7ff6ec0b28a0 con 0x7ff6ec13f710 2024-03-25T19:38:14.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.393+0000 7ff6ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:14.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.393+0000 7ff6ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:14.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.393+0000 7ff6ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:14.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.393+0000 7ff6ea7fc640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:38:14.393 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.393+0000 7ff6ea7fc640 1 -- 172.21.15.52:0/2818093067 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6ec0b2a30 con 0x7ff6ec13f710 2024-03-25T19:38:14.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 1 -- 172.21.15.52:0/2818093067 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 287770298 0 0) 0x7ff6d4003520 con 0x7ff6ec1430c0 2024-03-25T19:38:14.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 1 -- 172.21.15.52:0/2818093067 >> v1:172.21.15.179:6789/0 conn(0x7ff6ec13f710 legacy=0x7ff6ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 1 -- 172.21.15.52:0/2818093067 >> v1:172.21.15.52:6789/0 conn(0x7ff6ec0a98f0 legacy=0x7ff6ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 1 monclient: found mon.noname-b 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 1 -- 172.21.15.52:0/2818093067 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec0b44c0 con 0x7ff6ec1430c0 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 1 -- 172.21.15.52:0/2818093067 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff6e4001a10 con 0x7ff6ec1430c0 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6ea7fc640 10 monclient: dump: 2024-03-25T19:38:14.398 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:14.399 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:14.399 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:14.399 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:14.399 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:14.399 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:14.399 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:14.399 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:14.399 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:14.399 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:14.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6f2038640 5 monclient: authenticate success, global_id 4703 2024-03-25T19:38:14.399 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.397+0000 7ff6f2038640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.427+0000 7ff6ea7fc640 1 -- 172.21.15.52:0/2818093067 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6e4001b70 con 0x7ff6ec1430c0 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.427+0000 7ff6ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.427+0000 7ff6ea7fc640 1 -- 172.21.15.52:0/2818093067 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff6e4001f80 con 0x7ff6ec1430c0 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.427+0000 7ff6ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.427+0000 7ff6ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.427+0000 7ff6ea7fc640 10 monclient: dump: 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:14.427 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.427+0000 7ff6f2038640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:14.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.427+0000 7ff6f2038640 4 set_mon_vals no callback set 2024-03-25T19:38:14.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.428+0000 7ff6f2038640 10 monclient: shutdown 2024-03-25T19:38:14.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.428+0000 7ff6f2038640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:14.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.428+0000 7ff6f2038640 1 -- 172.21.15.52:0/2818093067 >> v1:172.21.15.52:6790/0 conn(0x7ff6ec1430c0 legacy=0x7ff6ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.428+0000 7ff6f2038640 1 -- 172.21.15.52:0/2818093067 shutdown_connections 2024-03-25T19:38:14.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.428+0000 7ff6f2038640 1 -- 172.21.15.52:0/2818093067 >> 172.21.15.52:0/2818093067 conn(0x7ff6ec0b7ea0 msgr2=0x7ff6ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:14.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.428+0000 7ff6f2038640 1 -- 172.21.15.52:0/2818093067 shutdown_connections 2024-03-25T19:38:14.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.428+0000 7ff6f2038640 1 -- 172.21.15.52:0/2818093067 wait complete. 2024-03-25T19:38:14.430 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.429+0000 7ff6f2038640 10 monclient: build_initial_monmap 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.429+0000 7ff6f2038640 10 monclient: monmap: 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.429+0000 7ff6f2038640 1 librados: starting msgr at 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.429+0000 7ff6f2038640 1 librados: starting objecter 2024-03-25T19:38:14.431 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.429+0000 7ff6f2038640 1 Processor -- start 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.429+0000 7ff6f2038640 1 -- start start 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.429+0000 7ff6f2038640 1 librados: setting wanted keys 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.429+0000 7ff6f2038640 1 librados: calling monclient init 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.429+0000 7ff6f2038640 10 monclient: init 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.430+0000 7ff6f2038640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.430+0000 7ff6f2038640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.430+0000 7ff6f2038640 10 monclient(hunting): picked mon.noname-c con 0x7ff6ec0a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.430+0000 7ff6f2038640 10 monclient(hunting): picked mon.noname-a con 0x7ff6ec13f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.430+0000 7ff6f2038640 10 monclient(hunting): picked mon.noname-b con 0x7ff6ec1430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.430+0000 7ff6f2038640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec1369a0 con 0x7ff6ec13f710 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.430+0000 7ff6f2038640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec1d8210 con 0x7ff6ec1430c0 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.430+0000 7ff6f2038640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6ec0abc30 con 0x7ff6ec0a98f0 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.430+0000 7ff6f2038640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:14.432 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.430+0000 7ff6f2038640 10 monclient(hunting): authenticate will time out at 6542.610287s 2024-03-25T19:38:14.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.439+0000 7ff6ebfff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7ff6ec1430c0 0x7ff6ec1e1c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:51216/0 (socket says 172.21.15.52:51216) 2024-03-25T19:38:14.439 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.439+0000 7ff6ebfff640 1 -- 172.21.15.52:0/665872699 learned_addr learned my addr 172.21.15.52:0/665872699 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:14.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.443+0000 7ff6d3fff640 10 client.?.objecter ms_handle_connect 0x7ff6ec1430c0 2024-03-25T19:38:14.442 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.443+0000 7ff6d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:14.444 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.445+0000 7ff6d3fff640 10 client.?.objecter ms_handle_connect 0x7ff6ec0a98f0 2024-03-25T19:38:14.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.445+0000 7ff6d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:14.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.453+0000 7ff6d3fff640 10 client.?.objecter ms_handle_connect 0x7ff6ec13f710 2024-03-25T19:38:14.452 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.453+0000 7ff6d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:14.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.456+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285839628 0 0) 0x7ff6ec1d8210 con 0x7ff6ec1430c0 2024-03-25T19:38:14.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.456+0000 7ff6d3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:14.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.456+0000 7ff6d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:14.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.456+0000 7ff6d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:14.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.456+0000 7ff6d3fff640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:38:14.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.456+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d8004eb0 con 0x7ff6ec1430c0 2024-03-25T19:38:14.469 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.470+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439451279 0 0) 0x7ff6ec0abc30 con 0x7ff6ec0a98f0 2024-03-25T19:38:14.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.470+0000 7ff6d3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:14.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.470+0000 7ff6d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:14.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.470+0000 7ff6d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:14.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.470+0000 7ff6d3fff640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:38:14.470 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.470+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6ec1d8210 con 0x7ff6ec0a98f0 2024-03-25T19:38:14.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.472+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4112615290 0 0) 0x7ff6d8004eb0 con 0x7ff6ec1430c0 2024-03-25T19:38:14.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.472+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6ec0abc30 con 0x7ff6ec1430c0 2024-03-25T19:38:14.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.473+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff6e4001ec0 con 0x7ff6ec1430c0 2024-03-25T19:38:14.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.473+0000 7ff6d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:14.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.473+0000 7ff6d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:14.472 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.473+0000 7ff6d3fff640 10 monclient(hunting): dump: 2024-03-25T19:38:14.472 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:14.472 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:14.472 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:14.472 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:14.473 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:14.473 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:14.473 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:14.473 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:14.473 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:14.473 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:14.473 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.473+0000 7ff6d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:14.475 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.476+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277809096 0 0) 0x7ff6ec0b2a30 con 0x7ff6ec13f710 2024-03-25T19:38:14.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.476+0000 7ff6d3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:14.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.476+0000 7ff6d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:14.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.476+0000 7ff6d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:14.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.476+0000 7ff6d3fff640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:38:14.476 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.476+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d8004eb0 con 0x7ff6ec13f710 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2146613585 0 0) 0x7ff6ec0abc30 con 0x7ff6ec1430c0 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 >> v1:172.21.15.52:6790/0 conn(0x7ff6ec0a98f0 legacy=0x7ff6ec1d7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 >> v1:172.21.15.52:6789/0 conn(0x7ff6ec13f710 legacy=0x7ff6ec138700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6d3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6d3fff640 1 monclient: found mon.b 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec137ad0 con 0x7ff6ec1430c0 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6d3fff640 10 monclient: _finish_auth 0 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6d3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6f2038640 5 monclient: authenticate success, global_id 4723 2024-03-25T19:38:14.481 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6f2038640 10 monclient: _renew_subs 2024-03-25T19:38:14.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6f2038640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:14.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6f2038640 1 -- 172.21.15.52:0/665872699 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ec138e10 con 0x7ff6ec1430c0 2024-03-25T19:38:14.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6f2038640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:14.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6f2038640 10 monclient: _renew_subs 2024-03-25T19:38:14.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6f2038640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:14.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6f2038640 1 -- 172.21.15.52:0/665872699 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ec137f90 con 0x7ff6ec1430c0 2024-03-25T19:38:14.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.480+0000 7ff6f2038640 1 librados: init done 2024-03-25T19:38:14.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.481+0000 7ff6f2038640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:14.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.481+0000 7ff6f2038640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:14.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.481+0000 7ff6f2038640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:14.482 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.481+0000 7ff6f2038640 1 -- 172.21.15.52:0/665872699 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ec0627b0 con 0x7ff6ec1430c0 2024-03-25T19:38:14.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.759+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6e4004e10 con 0x7ff6ec1430c0 2024-03-25T19:38:14.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.759+0000 7ff6d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:14.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.759+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff6e4005120 con 0x7ff6ec1430c0 2024-03-25T19:38:14.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.759+0000 7ff6d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:14.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.759+0000 7ff6d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:14.759 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.759+0000 7ff6d3fff640 10 monclient: dump: 2024-03-25T19:38:14.759 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:14.759 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:14.759 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:14.759 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:14.760 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:14.760 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:14.760 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:14.760 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:14.760 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:14.760 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:14.760 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.759+0000 7ff6e9ffb640 4 set_mon_vals no callback set 2024-03-25T19:38:14.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.760+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7ff6e401b0f0 con 0x7ff6ec1430c0 2024-03-25T19:38:14.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.760+0000 7ff6eb7fe640 10 monclient: get_auth_request con 0x7ff6d806de00 auth_method 0 2024-03-25T19:38:14.761 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.761+0000 7ff6d3fff640 10 client.4723.objecter ms_handle_connect 0x7ff6d806de00 2024-03-25T19:38:14.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.765+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7ff6e4054bf0 con 0x7ff6ec1430c0 2024-03-25T19:38:14.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.765+0000 7ff6d3fff640 10 client.4723.objecter ms_dispatch 0x7ff6ec000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:38:14.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.765+0000 7ff6d3fff640 3 client.4723.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:38:14.764 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.765+0000 7ff6d3fff640 3 client.4723.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:38:14.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.765+0000 7ff6d3fff640 20 client.4723.objecter dump_active .. 0 homeless 2024-03-25T19:38:14.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.766+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6e4050700 con 0x7ff6ec1430c0 2024-03-25T19:38:14.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.766+0000 7ff6d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:14.765 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:14.766+0000 7ff6d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:15.034 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.035+0000 7ff6f2038640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.035+0000 7ff6f2038640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.035+0000 7ff6f2038640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:15.035 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.035+0000 7ff6f2038640 1 -- 172.21.15.52:0/665872699 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6ec069f40 con 0x7ff6ec1430c0 2024-03-25T19:38:15.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.107+0000 7ff6d3fff640 1 -- 172.21.15.52:0/665872699 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7ff6e4052630 con 0x7ff6ec1430c0 2024-03-25T19:38:15.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.107+0000 7ff6d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:15.106 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.107+0000 7ff6d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:15.106 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:38:15.106 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:15.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 10 librados: watch_flush enter 2024-03-25T19:38:15.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 10 librados: watch_flush exit 2024-03-25T19:38:15.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 20 client.4723.objecter shutdown clearing up homeless session... 2024-03-25T19:38:15.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 10 client.4723.objecter successfully canceled tick 2024-03-25T19:38:15.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 1 -- 172.21.15.52:0/665872699 >> v1:172.21.15.52:6800/3967427147 conn(0x7ff6d806de00 legacy=0x7ff6d80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 10 monclient: shutdown 2024-03-25T19:38:15.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:15.109 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 1 -- 172.21.15.52:0/665872699 >> v1:172.21.15.179:6789/0 conn(0x7ff6ec1430c0 legacy=0x7ff6ec1e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:15.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 1 -- 172.21.15.52:0/665872699 shutdown_connections 2024-03-25T19:38:15.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 1 -- 172.21.15.52:0/665872699 >> 172.21.15.52:0/665872699 conn(0x7ff6ec0b7ea0 msgr2=0x7ff6ec13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:15.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 1 -- 172.21.15.52:0/665872699 shutdown_connections 2024-03-25T19:38:15.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.109+0000 7ff6f2038640 1 -- 172.21.15.52:0/665872699 wait complete. 2024-03-25T19:38:15.110 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:15.110+0000 7ff6f2038640 1 librados: shutdown 2024-03-25T19:38:15.121 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:18.265 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:11.863 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.271 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:11.870 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.305 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:11.903 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.320 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:11.919 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.331 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:11.930 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.334 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:11.876 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.337 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:11.879 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.357 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:11.899 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:18.358 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:11.957 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:21.926 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:22.042 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.043+0000 7f6af357b640 10 monclient: get_monmap_and_config 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.043+0000 7f6af357b640 10 monclient: build_initial_monmap 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.043+0000 7f6af357b640 10 monclient: monmap: 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:38:22.044132+0000 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:38:22.044132+0000 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:38:22.043 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.044+0000 7f6af357b640 1 Processor -- start 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.044+0000 7f6af357b640 1 -- start start 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.044+0000 7f6af357b640 10 monclient: init 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.045+0000 7f6af357b640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.045+0000 7f6af357b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.045+0000 7f6af357b640 10 monclient(hunting): picked mon.noname-b con 0x7f6aec0a98f0 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.045+0000 7f6af357b640 10 monclient(hunting): picked mon.noname-a con 0x7f6aec13f710 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.045+0000 7f6af357b640 10 monclient(hunting): picked mon.noname-c con 0x7f6aec1430c0 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.045+0000 7f6af357b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec142520 con 0x7f6aec13f710 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.045+0000 7f6af357b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec0b28a0 con 0x7f6aec1430c0 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.045+0000 7f6af357b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec0b2a30 con 0x7f6aec0a98f0 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.045+0000 7f6af357b640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:22.045 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.045+0000 7f6af357b640 10 monclient(hunting): authenticate will time out at 6550.225280s 2024-03-25T19:38:22.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.050+0000 7f6af1317640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f6aec0a98f0 0x7f6aec0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:33232/0 (socket says 172.21.15.52:33232) 2024-03-25T19:38:22.049 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.050+0000 7f6af1317640 1 -- 172.21.15.52:0/2325870649 learned_addr learned my addr 172.21.15.52:0/2325870649 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:22.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.059+0000 7f6ae3fff640 1 -- 172.21.15.52:0/2325870649 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928561695 0 0) 0x7f6aec0b2a30 con 0x7f6aec0a98f0 2024-03-25T19:38:22.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.059+0000 7f6ae3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.059+0000 7f6ae3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.059+0000 7f6ae3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.059+0000 7f6ae3fff640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:38:22.058 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.059+0000 7f6ae3fff640 1 -- 172.21.15.52:0/2325870649 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad8003520 con 0x7f6aec0a98f0 2024-03-25T19:38:22.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 1 -- 172.21.15.52:0/2325870649 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2608548256 0 0) 0x7f6ad8003520 con 0x7f6aec0a98f0 2024-03-25T19:38:22.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 1 -- 172.21.15.52:0/2325870649 >> v1:172.21.15.179:6789/0 conn(0x7f6aec1430c0 legacy=0x7f6aec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.065 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 1 -- 172.21.15.52:0/2325870649 >> v1:172.21.15.52:6789/0 conn(0x7f6aec13f710 legacy=0x7f6aec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 1 monclient: found mon.noname-b 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 1 -- 172.21.15.52:0/2325870649 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec0b44c0 con 0x7f6aec0a98f0 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 10 monclient: _finish_auth 0 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 1 -- 172.21.15.52:0/2325870649 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6ae8001690 con 0x7f6aec0a98f0 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6ae3fff640 10 monclient: dump: 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:22.066 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:22.067 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:22.067 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.067 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:22.067 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:22.067 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:22.067 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:22.067 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:22.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6af357b640 5 monclient: authenticate success, global_id 4709 2024-03-25T19:38:22.067 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.066+0000 7f6af357b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:22.070 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.071+0000 7f6ae3fff640 1 -- 172.21.15.52:0/2325870649 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ae8001860 con 0x7f6aec0a98f0 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.071+0000 7f6ae3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.071+0000 7f6ae3fff640 1 -- 172.21.15.52:0/2325870649 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6ae8001cb0 con 0x7f6aec0a98f0 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.071+0000 7f6af357b640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.071+0000 7f6af357b640 4 set_mon_vals no callback set 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.071+0000 7f6ae3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.071+0000 7f6ae3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.071+0000 7f6ae3fff640 10 monclient: dump: 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:22.071 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:22.072 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:22.072 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:22.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.072+0000 7f6af357b640 10 monclient: shutdown 2024-03-25T19:38:22.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.072+0000 7f6af357b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:22.072 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.072+0000 7f6af357b640 1 -- 172.21.15.52:0/2325870649 >> v1:172.21.15.52:6790/0 conn(0x7f6aec0a98f0 legacy=0x7f6aec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.072+0000 7f6af357b640 1 -- 172.21.15.52:0/2325870649 shutdown_connections 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.072+0000 7f6af357b640 1 -- 172.21.15.52:0/2325870649 >> 172.21.15.52:0/2325870649 conn(0x7f6aec0b7ea0 msgr2=0x7f6aec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.072+0000 7f6af357b640 1 -- 172.21.15.52:0/2325870649 shutdown_connections 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.072+0000 7f6af357b640 1 -- 172.21.15.52:0/2325870649 wait complete. 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.073+0000 7f6af357b640 10 monclient: build_initial_monmap 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.073+0000 7f6af357b640 10 monclient: monmap: 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:38:22.073 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.074+0000 7f6af357b640 1 librados: starting msgr at 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.074+0000 7f6af357b640 1 librados: starting objecter 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.074+0000 7f6af357b640 1 Processor -- start 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.074+0000 7f6af357b640 1 -- start start 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.074+0000 7f6af357b640 1 librados: setting wanted keys 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.074+0000 7f6af357b640 1 librados: calling monclient init 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.074+0000 7f6af357b640 10 monclient: init 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.074+0000 7f6af357b640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.074+0000 7f6af357b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.074+0000 7f6af357b640 10 monclient(hunting): picked mon.noname-a con 0x7f6aec0a98f0 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.075+0000 7f6af357b640 10 monclient(hunting): picked mon.noname-b con 0x7f6aec13f710 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.075+0000 7f6af357b640 10 monclient(hunting): picked mon.noname-c con 0x7f6aec1430c0 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.075+0000 7f6af357b640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec137f40 con 0x7f6aec0a98f0 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.075+0000 7f6af357b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec0abfa0 con 0x7f6aec13f710 2024-03-25T19:38:22.075 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.075+0000 7f6af357b640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aec0ac280 con 0x7f6aec1430c0 2024-03-25T19:38:22.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.075+0000 7f6af357b640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:22.076 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.075+0000 7f6af357b640 10 monclient(hunting): authenticate will time out at 6550.255101s 2024-03-25T19:38:22.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.081+0000 7f6af0b16640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f6aec13f710 0x7f6aec138c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:47436/0 (socket says 172.21.15.52:47436) 2024-03-25T19:38:22.080 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.081+0000 7f6af0b16640 1 -- 172.21.15.52:0/3313871207 learned_addr learned my addr 172.21.15.52:0/3313871207 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:22.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.087+0000 7f6ae1ffb640 10 client.?.objecter ms_handle_connect 0x7f6aec13f710 2024-03-25T19:38:22.086 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.087+0000 7f6ae1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.089+0000 7f6ae1ffb640 10 client.?.objecter ms_handle_connect 0x7f6aec0a98f0 2024-03-25T19:38:22.088 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.089+0000 7f6ae1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.097+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030194692 0 0) 0x7f6aec137f40 con 0x7f6aec0a98f0 2024-03-25T19:38:22.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.097+0000 7f6ae1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.097+0000 7f6ae1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.097+0000 7f6ae1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.097+0000 7f6ae1ffb640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:38:22.096 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.097+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad0004eb0 con 0x7f6aec0a98f0 2024-03-25T19:38:22.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.098+0000 7f6ae1ffb640 10 client.?.objecter ms_handle_connect 0x7f6aec1430c0 2024-03-25T19:38:22.097 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.098+0000 7f6ae1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:22.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.103+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087808769 0 0) 0x7f6aec0abfa0 con 0x7f6aec13f710 2024-03-25T19:38:22.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.103+0000 7f6ae1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.103+0000 7f6ae1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.103+0000 7f6ae1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.103+0000 7f6ae1ffb640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:38:22.103 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.103+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aec137f40 con 0x7f6aec13f710 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.105+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786145805 0 0) 0x7f6aec0ac280 con 0x7f6aec1430c0 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.105+0000 7f6ae1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.105+0000 7f6ae1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.105+0000 7f6ae1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.105+0000 7f6ae1ffb640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:38:22.105 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.105+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aec0abfa0 con 0x7f6aec1430c0 2024-03-25T19:38:22.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 11559452 0 0) 0x7f6aec137f40 con 0x7f6aec13f710 2024-03-25T19:38:22.120 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6aec0ac280 con 0x7f6aec13f710 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6adc0018e0 con 0x7f6aec13f710 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:22.121 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1701439217 0 0) 0x7f6ad0004eb0 con 0x7f6aec0a98f0 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6aec137f40 con 0x7f6aec0a98f0 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6ae8001b70 con 0x7f6aec0a98f0 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.121+0000 7f6ae1ffb640 10 monclient(hunting): dump: 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:22.122 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.137+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2999926461 0 0) 0x7f6aec137f40 con 0x7f6aec0a98f0 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 >> v1:172.21.15.52:6790/0 conn(0x7f6aec1430c0 legacy=0x7f6aec1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 >> v1:172.21.15.179:6789/0 conn(0x7f6aec13f710 legacy=0x7f6aec138c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6ae1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6ae1ffb640 1 monclient: found mon.a 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6ae1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6ae1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec138190 con 0x7f6aec0a98f0 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6ae1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6ae1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6ae1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6af357b640 5 monclient: authenticate success, global_id 4713 2024-03-25T19:38:22.138 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6af357b640 10 monclient: _renew_subs 2024-03-25T19:38:22.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6af357b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:38:22.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6af357b640 1 -- 172.21.15.52:0/3313871207 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aec0acdb0 con 0x7f6aec0a98f0 2024-03-25T19:38:22.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6af357b640 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:22.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6af357b640 10 monclient: _renew_subs 2024-03-25T19:38:22.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6af357b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:38:22.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6af357b640 1 -- 172.21.15.52:0/3313871207 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aec139540 con 0x7f6aec0a98f0 2024-03-25T19:38:22.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.138+0000 7f6af357b640 1 librados: init done 2024-03-25T19:38:22.139 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.139+0000 7f6af357b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:22.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.139+0000 7f6af357b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:22.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.139+0000 7f6af357b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:38:22.140 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.139+0000 7f6af357b640 1 -- 172.21.15.52:0/3313871207 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aec0627b0 con 0x7f6aec0a98f0 2024-03-25T19:38:22.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.448+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ae8001010 con 0x7f6aec0a98f0 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.448+0000 7f6ae1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.448+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f6ae8004bf0 con 0x7f6aec0a98f0 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.448+0000 7f6ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.448+0000 7f6ae1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.448+0000 7f6ae1ffb640 10 monclient: dump: 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:22.448 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:22.449 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:22.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.448+0000 7f6ae37fe640 4 set_mon_vals no callback set 2024-03-25T19:38:22.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.449+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f6ae801abc0 con 0x7f6aec0a98f0 2024-03-25T19:38:22.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.450+0000 7f6af0b16640 10 monclient: get_auth_request con 0x7f6ad006df60 auth_method 0 2024-03-25T19:38:22.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.450+0000 7f6ae1ffb640 10 client.4713.objecter ms_handle_connect 0x7f6ad006df60 2024-03-25T19:38:22.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.455+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7f6ae8054860 con 0x7f6aec0a98f0 2024-03-25T19:38:22.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.455+0000 7f6ae1ffb640 10 client.4713.objecter ms_dispatch 0x7f6aec000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:38:22.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.455+0000 7f6ae1ffb640 3 client.4713.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:38:22.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.455+0000 7f6ae1ffb640 3 client.4713.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:38:22.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.455+0000 7f6ae1ffb640 20 client.4713.objecter dump_active .. 0 homeless 2024-03-25T19:38:22.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.455+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ae80501a0 con 0x7f6aec0a98f0 2024-03-25T19:38:22.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.455+0000 7f6ae1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:22.455 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.455+0000 7f6ae1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:22.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.694+0000 7f6af357b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:22.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.694+0000 7f6af357b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:22.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.694+0000 7f6af357b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:38:22.694 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.694+0000 7f6af357b640 1 -- 172.21.15.52:0/3313871207 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6aec141a00 con 0x7f6aec0a98f0 2024-03-25T19:38:22.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.724+0000 7f6ae1ffb640 1 -- 172.21.15.52:0/3313871207 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7f6ae80522a0 con 0x7f6aec0a98f0 2024-03-25T19:38:22.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.724+0000 7f6ae1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:22.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.724+0000 7f6ae1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:22.724 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:38:22.724 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 10 librados: watch_flush enter 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 10 librados: watch_flush exit 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 20 client.4713.objecter shutdown clearing up homeless session... 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 10 client.4713.objecter successfully canceled tick 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 1 -- 172.21.15.52:0/3313871207 >> v1:172.21.15.52:6800/3967427147 conn(0x7f6ad006df60 legacy=0x7f6ad0070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 10 monclient: shutdown 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 1 -- 172.21.15.52:0/3313871207 >> v1:172.21.15.52:6789/0 conn(0x7f6aec0a98f0 legacy=0x7f6aec0ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 1 -- 172.21.15.52:0/3313871207 shutdown_connections 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 1 -- 172.21.15.52:0/3313871207 >> 172.21.15.52:0/3313871207 conn(0x7f6aec0b7ea0 msgr2=0x7f6aec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 1 -- 172.21.15.52:0/3313871207 shutdown_connections 2024-03-25T19:38:22.727 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 1 -- 172.21.15.52:0/3313871207 wait complete. 2024-03-25T19:38:22.728 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:22.727+0000 7f6af357b640 1 librados: shutdown 2024-03-25T19:38:22.738 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:26.997 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:20.596 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:27.002 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:20.601 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:27.008 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:20.607 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:27.011 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:20.610 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:27.012 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:20.554 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:27.015 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:20.558 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:27.024 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:20.623 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:27.032 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:20.574 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:27.091 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:20.690 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:29.543 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:29.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.659+0000 7f7392c4e640 10 monclient: get_monmap_and_config 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.659+0000 7f7392c4e640 10 monclient: build_initial_monmap 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.659+0000 7f7392c4e640 10 monclient: monmap: 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:38:29.660174+0000 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:38:29.660174+0000 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:38:29.659 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:29.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.660+0000 7f7392c4e640 1 Processor -- start 2024-03-25T19:38:29.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.660+0000 7f7392c4e640 1 -- start start 2024-03-25T19:38:29.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.660+0000 7f7392c4e640 10 monclient: init 2024-03-25T19:38:29.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.661+0000 7f7392c4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:29.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.661+0000 7f7392c4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:29.660 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.661+0000 7f7392c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f738c0b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:29.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.661+0000 7f7392c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f738c0b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:29.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.661+0000 7f7392c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f738c14be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:29.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.661+0000 7f7392c4e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c0af940 con 0x7f738c14be90 2024-03-25T19:38:29.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.661+0000 7f7392c4e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c0b0a70 con 0x7f738c0b1730 2024-03-25T19:38:29.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.661+0000 7f7392c4e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c0b0d50 con 0x7f738c0b4d60 2024-03-25T19:38:29.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.661+0000 7f7392c4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:29.661 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.661+0000 7f7392c4e640 10 monclient(hunting): authenticate will time out at 6557.841374s 2024-03-25T19:38:29.672 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.673+0000 7f738bfff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f738c0b1730 0x7f738c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:45964/0 (socket says 172.21.15.52:45964) 2024-03-25T19:38:29.672 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.673+0000 7f738bfff640 1 -- 172.21.15.52:0/1194723375 learned_addr learned my addr 172.21.15.52:0/1194723375 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:29.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.685+0000 7f738b7fe640 1 -- 172.21.15.52:0/1194723375 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057170253 0 0) 0x7f738c0b0a70 con 0x7f738c0b1730 2024-03-25T19:38:29.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.685+0000 7f738b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.685+0000 7f738b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.685+0000 7f738b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.685+0000 7f738b7fe640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:38:29.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.685+0000 7f738b7fe640 1 -- 172.21.15.52:0/1194723375 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f737c003520 con 0x7f738c0b1730 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 1 -- 172.21.15.52:0/1194723375 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 452251250 0 0) 0x7f737c003520 con 0x7f738c0b1730 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 1 -- 172.21.15.52:0/1194723375 >> v1:172.21.15.52:6790/0 conn(0x7f738c0b4d60 legacy=0x7f738c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 1 -- 172.21.15.52:0/1194723375 >> v1:172.21.15.52:6789/0 conn(0x7f738c14be90 legacy=0x7f738c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 1 -- 172.21.15.52:0/1194723375 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c0b1030 con 0x7f738c0b1730 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:29.698 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f7392c4e640 5 monclient: authenticate success, global_id 4732 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f7392c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 1 -- 172.21.15.52:0/1194723375 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7378001690 con 0x7f738c0b1730 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f738b7fe640 10 monclient: dump: 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:29.699 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.698+0000 7f7392c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:29.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.710+0000 7f738b7fe640 1 -- 172.21.15.52:0/1194723375 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7378001920 con 0x7f738c0b1730 2024-03-25T19:38:29.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.710+0000 7f738b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:29.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.710+0000 7f738b7fe640 1 -- 172.21.15.52:0/1194723375 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7378001690 con 0x7f738c0b1730 2024-03-25T19:38:29.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.710+0000 7f738b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.710+0000 7f738b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:29.709 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.710+0000 7f738b7fe640 10 monclient: dump: 2024-03-25T19:38:29.709 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:29.709 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:29.709 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:29.709 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:29.709 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.710 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:29.710 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:29.710 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:29.710 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:29.710 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:29.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.710+0000 7f7392c4e640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:29.710 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.710+0000 7f7392c4e640 4 set_mon_vals no callback set 2024-03-25T19:38:29.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.710+0000 7f7392c4e640 10 monclient: shutdown 2024-03-25T19:38:29.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.710+0000 7f7392c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:29.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.710+0000 7f7392c4e640 1 -- 172.21.15.52:0/1194723375 >> v1:172.21.15.179:6789/0 conn(0x7f738c0b1730 legacy=0x7f738c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.711+0000 7f7392c4e640 1 -- 172.21.15.52:0/1194723375 shutdown_connections 2024-03-25T19:38:29.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.711+0000 7f7392c4e640 1 -- 172.21.15.52:0/1194723375 >> 172.21.15.52:0/1194723375 conn(0x7f738c0617a0 msgr2=0x7f738c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:29.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.711+0000 7f7392c4e640 1 -- 172.21.15.52:0/1194723375 shutdown_connections 2024-03-25T19:38:29.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.711+0000 7f7392c4e640 1 -- 172.21.15.52:0/1194723375 wait complete. 2024-03-25T19:38:29.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.712+0000 7f7392c4e640 10 monclient: build_initial_monmap 2024-03-25T19:38:29.711 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.712+0000 7f7392c4e640 10 monclient: monmap: 2024-03-25T19:38:29.711 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:29.711 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:29.712 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:38:29.712 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:38:29.712 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:29.712 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:38:29.712 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:29.712 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:38:29.712 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:38:29.712 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.712+0000 7f7392c4e640 1 librados: starting msgr at 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.712+0000 7f7392c4e640 1 librados: starting objecter 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.712+0000 7f7392c4e640 1 Processor -- start 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.712+0000 7f7392c4e640 1 -- start start 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.712+0000 7f7392c4e640 1 librados: setting wanted keys 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.712+0000 7f7392c4e640 1 librados: calling monclient init 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.712+0000 7f7392c4e640 10 monclient: init 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.713+0000 7f7392c4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.714+0000 7f7392c4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.714+0000 7f7392c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f738c0b1730 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.714+0000 7f7392c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f738c0b4d60 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.714+0000 7f7392c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f738c14be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.714+0000 7f7392c4e640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c143020 con 0x7f738c14be90 2024-03-25T19:38:29.714 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.714+0000 7f7392c4e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c141d20 con 0x7f738c0b1730 2024-03-25T19:38:29.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.714+0000 7f7392c4e640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c141eb0 con 0x7f738c0b4d60 2024-03-25T19:38:29.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.714+0000 7f7392c4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:29.715 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.714+0000 7f7392c4e640 10 monclient(hunting): authenticate will time out at 6557.894231s 2024-03-25T19:38:29.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.717+0000 7f738bfff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f738c0b4d60 0x7f738c132d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:45174/0 (socket says 172.21.15.52:45174) 2024-03-25T19:38:29.716 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.717+0000 7f738bfff640 1 -- 172.21.15.52:0/3031334772 learned_addr learned my addr 172.21.15.52:0/3031334772 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:29.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.718+0000 7f73897fa640 10 client.?.objecter ms_handle_connect 0x7f738c0b4d60 2024-03-25T19:38:29.717 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.718+0000 7f73897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:29.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.723+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487021353 0 0) 0x7f738c141eb0 con 0x7f738c0b4d60 2024-03-25T19:38:29.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.723+0000 7f73897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.723+0000 7f73897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.723+0000 7f73897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.723+0000 7f73897fa640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:38:29.723 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.723+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7370004eb0 con 0x7f738c0b4d60 2024-03-25T19:38:29.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.725+0000 7f73897fa640 10 client.?.objecter ms_handle_connect 0x7f738c0b1730 2024-03-25T19:38:29.724 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.725+0000 7f73897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:29.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.735+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967883521 0 0) 0x7f738c141d20 con 0x7f738c0b1730 2024-03-25T19:38:29.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.735+0000 7f73897fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:29.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.735+0000 7f73897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:29.734 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.735+0000 7f73897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:29.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.735+0000 7f73897fa640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:38:29.735 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.735+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f738c141eb0 con 0x7f738c0b1730 2024-03-25T19:38:29.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.743+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2496697723 0 0) 0x7f738c141eb0 con 0x7f738c0b1730 2024-03-25T19:38:29.742 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.743+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f738c141d20 con 0x7f738c0b1730 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.743+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7380001510 con 0x7f738c0b1730 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.743+0000 7f73897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.743+0000 7f73897fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.743+0000 7f73897fa640 10 monclient(hunting): dump: 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:29.743 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.743+0000 7f73897fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:29.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.744+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 324944412 0 0) 0x7f7370004eb0 con 0x7f738c0b4d60 2024-03-25T19:38:29.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.744+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f738c141eb0 con 0x7f738c0b4d60 2024-03-25T19:38:29.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.744+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f73780047e0 con 0x7f738c0b4d60 2024-03-25T19:38:29.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.744+0000 7f73897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:29.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.744+0000 7f73897fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:29.744 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.744+0000 7f73897fa640 10 monclient(hunting): dump: 2024-03-25T19:38:29.744 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:29.744 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:29.745 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:29.745 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:29.745 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:29.745 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:29.745 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:29.745 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:29.745 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:29.745 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:29.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.752+0000 7f73897fa640 10 client.?.objecter ms_handle_connect 0x7f738c14be90 2024-03-25T19:38:29.751 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.752+0000 7f73897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:29.753 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 366678871 0 0) 0x7f738c141d20 con 0x7f738c0b1730 2024-03-25T19:38:29.753 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 >> v1:172.21.15.52:6790/0 conn(0x7f738c0b4d60 legacy=0x7f738c132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 >> v1:172.21.15.52:6789/0 conn(0x7f738c14be90 legacy=0x7f738c140580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f73897fa640 10 monclient: _finish_hunting 0 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f73897fa640 1 monclient: found mon.b 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f73897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f73897fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c13e1f0 con 0x7f738c0b1730 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f73897fa640 10 monclient: _finish_auth 0 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f73897fa640 10 monclient: _check_auth_tickets 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f73897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f7392c4e640 5 monclient: authenticate success, global_id 4735 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f7392c4e640 10 monclient: _renew_subs 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f7392c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f7392c4e640 1 -- 172.21.15.52:0/3031334772 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738c13e440 con 0x7f738c0b1730 2024-03-25T19:38:29.754 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f7392c4e640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:29.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f7392c4e640 10 monclient: _renew_subs 2024-03-25T19:38:29.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f7392c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:29.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f7392c4e640 1 -- 172.21.15.52:0/3031334772 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f738c133610 con 0x7f738c0b1730 2024-03-25T19:38:29.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.754+0000 7f7392c4e640 1 librados: init done 2024-03-25T19:38:29.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.755+0000 7f7392c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:29.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.755+0000 7f7392c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:29.755 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.755+0000 7f7392c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:29.756 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:29.755+0000 7f7392c4e640 1 -- 172.21.15.52:0/3031334772 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738c0b2270 con 0x7f738c0b1730 2024-03-25T19:38:30.053 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.054+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7380001a20 con 0x7f738c0b1730 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.054+0000 7f73897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.054+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f7380003910 con 0x7f738c0b1730 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.054+0000 7f73897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.054+0000 7f73897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.054+0000 7f73897fa640 10 monclient: dump: 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:30.054 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:30.055 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:30.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.054+0000 7f738affd640 4 set_mon_vals no callback set 2024-03-25T19:38:30.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.055+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f73800198e0 con 0x7f738c0b1730 2024-03-25T19:38:30.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.055+0000 7f738bfff640 10 monclient: get_auth_request con 0x7f737006ddf0 auth_method 0 2024-03-25T19:38:30.055 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.056+0000 7f73897fa640 10 client.4735.objecter ms_handle_connect 0x7f737006ddf0 2024-03-25T19:38:30.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.060+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7f7380053500 con 0x7f738c0b1730 2024-03-25T19:38:30.059 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.060+0000 7f73897fa640 10 client.4735.objecter ms_dispatch 0x7f738c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:38:30.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.060+0000 7f73897fa640 3 client.4735.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:38:30.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.060+0000 7f73897fa640 3 client.4735.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:38:30.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.060+0000 7f73897fa640 20 client.4735.objecter dump_active .. 0 homeless 2024-03-25T19:38:30.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.060+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f738004f010 con 0x7f738c0b1730 2024-03-25T19:38:30.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.060+0000 7f73897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:30.060 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.060+0000 7f73897fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:30.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.323+0000 7f7392c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:30.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.323+0000 7f7392c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:30.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.323+0000 7f7392c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:30.323 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.323+0000 7f7392c4e640 1 -- 172.21.15.52:0/3031334772 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f738c0627b0 con 0x7f738c0b1730 2024-03-25T19:38:30.356 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.357+0000 7f73897fa640 1 -- 172.21.15.52:0/3031334772 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7f7380050f40 con 0x7f738c0b1730 2024-03-25T19:38:30.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.357+0000 7f73897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:30.357 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.357+0000 7f73897fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:30.357 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:38:30.357 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:30.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.359+0000 7f7392c4e640 10 librados: watch_flush enter 2024-03-25T19:38:30.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.359+0000 7f7392c4e640 10 librados: watch_flush exit 2024-03-25T19:38:30.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.359+0000 7f7392c4e640 20 client.4735.objecter shutdown clearing up homeless session... 2024-03-25T19:38:30.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.359+0000 7f7392c4e640 10 client.4735.objecter successfully canceled tick 2024-03-25T19:38:30.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.359+0000 7f7392c4e640 1 -- 172.21.15.52:0/3031334772 >> v1:172.21.15.52:6800/3967427147 conn(0x7f737006ddf0 legacy=0x7f73700702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.359 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.359+0000 7f7392c4e640 10 monclient: shutdown 2024-03-25T19:38:30.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.359+0000 7f7392c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:30.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.359+0000 7f7392c4e640 1 -- 172.21.15.52:0/3031334772 >> v1:172.21.15.179:6789/0 conn(0x7f738c0b1730 legacy=0x7f738c142660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:30.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.359+0000 7f7392c4e640 1 -- 172.21.15.52:0/3031334772 shutdown_connections 2024-03-25T19:38:30.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.360+0000 7f7392c4e640 1 -- 172.21.15.52:0/3031334772 >> 172.21.15.52:0/3031334772 conn(0x7f738c0617a0 msgr2=0x7f738c0b2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:30.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.360+0000 7f7392c4e640 1 -- 172.21.15.52:0/3031334772 shutdown_connections 2024-03-25T19:38:30.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.360+0000 7f7392c4e640 1 -- 172.21.15.52:0/3031334772 wait complete. 2024-03-25T19:38:30.360 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:30.360+0000 7f7392c4e640 1 librados: shutdown 2024-03-25T19:38:30.371 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:37.177 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:37.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.293+0000 7ff8ea674640 10 monclient: get_monmap_and_config 2024-03-25T19:38:37.292 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.293+0000 7ff8ea674640 10 monclient: build_initial_monmap 2024-03-25T19:38:37.293 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.293+0000 7ff8ea674640 10 monclient: monmap: 2024-03-25T19:38:37.293 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:37.293 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:37.293 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:38:37.294115+0000 2024-03-25T19:38:37.293 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:38:37.294115+0000 2024-03-25T19:38:37.293 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:37.293 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:37.293 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:37.293 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:38:37.293 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:38:37.293 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:37.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.294+0000 7ff8ea674640 1 Processor -- start 2024-03-25T19:38:37.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.294+0000 7ff8ea674640 1 -- start start 2024-03-25T19:38:37.294 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.294+0000 7ff8ea674640 10 monclient: init 2024-03-25T19:38:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.295+0000 7ff8ea674640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.295+0000 7ff8ea674640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.295+0000 7ff8ea674640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e40b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.295+0000 7ff8ea674640 10 monclient(hunting): picked mon.noname-a con 0x7ff8e40b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.295+0000 7ff8ea674640 10 monclient(hunting): picked mon.noname-b con 0x7ff8e414be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.295+0000 7ff8ea674640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e40af940 con 0x7ff8e40b1730 2024-03-25T19:38:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.295+0000 7ff8ea674640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e40b0a70 con 0x7ff8e40b4d60 2024-03-25T19:38:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.295+0000 7ff8ea674640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e40b0d50 con 0x7ff8e414be90 2024-03-25T19:38:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.295+0000 7ff8ea674640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:37.295 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.295+0000 7ff8ea674640 10 monclient(hunting): authenticate will time out at 6565.475327s 2024-03-25T19:38:37.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.314+0000 7ff8e3fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7ff8e40b4d60 0x7ff8e413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:36684/0 (socket says 172.21.15.52:36684) 2024-03-25T19:38:37.313 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.314+0000 7ff8e3fff640 1 -- 172.21.15.52:0/1444219411 learned_addr learned my addr 172.21.15.52:0/1444219411 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:37.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.327+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208180903 0 0) 0x7ff8e40b0a70 con 0x7ff8e40b4d60 2024-03-25T19:38:37.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.328+0000 7ff8e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.328+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.328+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.328+0000 7ff8e2ffd640 10 monclient(hunting): my global_id is 4738 2024-03-25T19:38:37.327 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.328+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8cc003520 con 0x7ff8e40b4d60 2024-03-25T19:38:37.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.333+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808735105 0 0) 0x7ff8e40af940 con 0x7ff8e40b1730 2024-03-25T19:38:37.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.333+0000 7ff8e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.333+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.333+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.333+0000 7ff8e2ffd640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:38:37.333 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.333+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e40b0a70 con 0x7ff8e40b1730 2024-03-25T19:38:37.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.346+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824071858 0 0) 0x7ff8e40b0d50 con 0x7ff8e414be90 2024-03-25T19:38:37.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.346+0000 7ff8e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.346+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.346+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.346+0000 7ff8e2ffd640 10 monclient(hunting): my global_id is 4718 2024-03-25T19:38:37.345 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.346+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e40af940 con 0x7ff8e414be90 2024-03-25T19:38:37.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 514893888 0 0) 0x7ff8e40b0a70 con 0x7ff8e40b1730 2024-03-25T19:38:37.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 >> v1:172.21.15.52:6790/0 conn(0x7ff8e414be90 legacy=0x7ff8e414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 >> v1:172.21.15.179:6789/0 conn(0x7ff8e40b4d60 legacy=0x7ff8e413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:37.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 1 monclient: found mon.noname-a 2024-03-25T19:38:37.347 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.52:6789/0 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e40b1030 con 0x7ff8e40b1730 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff8d8001670 con 0x7ff8e40b1730 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8ea674640 5 monclient: authenticate success, global_id 4722 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8ea674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8e2ffd640 10 monclient: dump: 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.348 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.349 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.349 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:37.349 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:37.349 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:37.349 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:37.349 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:37.349 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.347+0000 7ff8ea674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:37.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.385+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 <== mon.0 v1:172.21.15.52:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8d80017d0 con 0x7ff8e40b1730 2024-03-25T19:38:37.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.385+0000 7ff8e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:37.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.385+0000 7ff8e2ffd640 1 -- 172.21.15.52:0/1444219411 <== mon.0 v1:172.21.15.52:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff8d8001be0 con 0x7ff8e40b1730 2024-03-25T19:38:37.384 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.385+0000 7ff8ea674640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.385+0000 7ff8ea674640 4 set_mon_vals no callback set 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.385+0000 7ff8e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.385+0000 7ff8e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.385+0000 7ff8e2ffd640 10 monclient: dump: 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:37.385 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.386+0000 7ff8ea674640 10 monclient: shutdown 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.386+0000 7ff8ea674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.386+0000 7ff8ea674640 1 -- 172.21.15.52:0/1444219411 >> v1:172.21.15.52:6789/0 conn(0x7ff8e40b1730 legacy=0x7ff8e40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.386+0000 7ff8ea674640 1 -- 172.21.15.52:0/1444219411 shutdown_connections 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.386+0000 7ff8ea674640 1 -- 172.21.15.52:0/1444219411 >> 172.21.15.52:0/1444219411 conn(0x7ff8e40617a0 msgr2=0x7ff8e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.386+0000 7ff8ea674640 1 -- 172.21.15.52:0/1444219411 shutdown_connections 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.386+0000 7ff8ea674640 1 -- 172.21.15.52:0/1444219411 wait complete. 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.386+0000 7ff8ea674640 10 monclient: build_initial_monmap 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.386+0000 7ff8ea674640 10 monclient: monmap: 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:37.387 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.387+0000 7ff8ea674640 1 librados: starting msgr at 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.387+0000 7ff8ea674640 1 librados: starting objecter 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.387+0000 7ff8ea674640 1 Processor -- start 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.387+0000 7ff8ea674640 1 -- start start 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.387+0000 7ff8ea674640 1 librados: setting wanted keys 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.387+0000 7ff8ea674640 1 librados: calling monclient init 2024-03-25T19:38:37.388 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.387+0000 7ff8ea674640 10 monclient: init 2024-03-25T19:38:37.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.388+0000 7ff8ea674640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:37.389 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.388+0000 7ff8ea674640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:37.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.388+0000 7ff8ea674640 10 monclient(hunting): picked mon.noname-a con 0x7ff8e40b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:37.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.388+0000 7ff8ea674640 10 monclient(hunting): picked mon.noname-b con 0x7ff8e40b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:37.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.389+0000 7ff8ea674640 10 monclient(hunting): picked mon.noname-c con 0x7ff8e414be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:37.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.389+0000 7ff8ea674640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e413d0f0 con 0x7ff8e40b1730 2024-03-25T19:38:37.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.389+0000 7ff8ea674640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e4140380 con 0x7ff8e40b4d60 2024-03-25T19:38:37.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.389+0000 7ff8ea674640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e4141820 con 0x7ff8e414be90 2024-03-25T19:38:37.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.389+0000 7ff8ea674640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:37.390 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.389+0000 7ff8ea674640 10 monclient(hunting): authenticate will time out at 6565.569101s 2024-03-25T19:38:37.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.394+0000 7ff8e8c11640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7ff8e414be90 0x7ff8e41dae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:59278/0 (socket says 172.21.15.52:59278) 2024-03-25T19:38:37.394 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.394+0000 7ff8e8c11640 1 -- 172.21.15.52:0/2270572202 learned_addr learned my addr 172.21.15.52:0/2270572202 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:37.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.398+0000 7ff8e0ff9640 10 client.?.objecter ms_handle_connect 0x7ff8e40b4d60 2024-03-25T19:38:37.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.398+0000 7ff8e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:37.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.398+0000 7ff8e0ff9640 10 client.?.objecter ms_handle_connect 0x7ff8e414be90 2024-03-25T19:38:37.397 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.398+0000 7ff8e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:37.403 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.404+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93454486 0 0) 0x7ff8e4140380 con 0x7ff8e40b4d60 2024-03-25T19:38:37.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.404+0000 7ff8e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.404+0000 7ff8e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.404+0000 7ff8e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.404+0000 7ff8e0ff9640 10 monclient(hunting): my global_id is 4741 2024-03-25T19:38:37.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.404+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d0004eb0 con 0x7ff8e40b4d60 2024-03-25T19:38:37.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.405+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50662088 0 0) 0x7ff8e4141820 con 0x7ff8e414be90 2024-03-25T19:38:37.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.405+0000 7ff8e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.404 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.405+0000 7ff8e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.405+0000 7ff8e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.405+0000 7ff8e0ff9640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:38:37.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.405+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e4140380 con 0x7ff8e414be90 2024-03-25T19:38:37.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.406+0000 7ff8e0ff9640 10 client.?.objecter ms_handle_connect 0x7ff8e40b1730 2024-03-25T19:38:37.405 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.406+0000 7ff8e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:37.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.415+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439218703 0 0) 0x7ff8e413d0f0 con 0x7ff8e40b1730 2024-03-25T19:38:37.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.415+0000 7ff8e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:37.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.415+0000 7ff8e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:37.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.415+0000 7ff8e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:37.414 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.415+0000 7ff8e0ff9640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:38:37.415 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.415+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e4141820 con 0x7ff8e40b1730 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.416+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 310387286 0 0) 0x7ff8e4140380 con 0x7ff8e414be90 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.416+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8e413d0f0 con 0x7ff8e414be90 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.416+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff8dc001c50 con 0x7ff8e414be90 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.416+0000 7ff8e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.416+0000 7ff8e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.416+0000 7ff8e0ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.416 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:37.417 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:37.417 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:37.417 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:37.417 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:37.417 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.416+0000 7ff8e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:37.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.429+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091688500 0 0) 0x7ff8d0004eb0 con 0x7ff8e40b4d60 2024-03-25T19:38:37.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.429+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8e4140380 con 0x7ff8e40b4d60 2024-03-25T19:38:37.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.429+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff8d8001ba0 con 0x7ff8e40b4d60 2024-03-25T19:38:37.428 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.429+0000 7ff8e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.429+0000 7ff8e0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.429+0000 7ff8e0ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:37.429 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.445+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3566877814 0 0) 0x7ff8e4141820 con 0x7ff8e40b1730 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.445+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8d0004eb0 con 0x7ff8e40b1730 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.445+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff8d4001910 con 0x7ff8e40b1730 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.445+0000 7ff8e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.445+0000 7ff8e0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.445+0000 7ff8e0ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.445 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:37.446 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:37.446 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:37.446 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:37.446 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 554216530 0 0) 0x7ff8e4140380 con 0x7ff8e40b4d60 2024-03-25T19:38:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 >> v1:172.21.15.52:6790/0 conn(0x7ff8e414be90 legacy=0x7ff8e41dae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 >> v1:172.21.15.52:6789/0 conn(0x7ff8e40b1730 legacy=0x7ff8e4141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8e0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:38:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8e0ff9640 1 monclient: found mon.b 2024-03-25T19:38:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:37.447 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8e0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e41419b0 con 0x7ff8e40b4d60 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8e0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8e0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8ea674640 5 monclient: authenticate success, global_id 4741 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8ea674640 10 monclient: _renew_subs 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8ea674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8ea674640 1 -- 172.21.15.52:0/2270572202 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e4143140 con 0x7ff8e40b4d60 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8ea674640 10 client.4741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8ea674640 10 monclient: _renew_subs 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8ea674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8ea674640 1 -- 172.21.15.52:0/2270572202 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e4133680 con 0x7ff8e40b4d60 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.446+0000 7ff8ea674640 1 librados: init done 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.447+0000 7ff8ea674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.447+0000 7ff8ea674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:37.448 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.447+0000 7ff8ea674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:37.449 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.447+0000 7ff8ea674640 1 -- 172.21.15.52:0/2270572202 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e4141b40 con 0x7ff8e40b4d60 2024-03-25T19:38:37.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.745+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8d80038a0 con 0x7ff8e40b4d60 2024-03-25T19:38:37.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.745+0000 7ff8e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:37.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.745+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7ff8d8003bb0 con 0x7ff8e40b4d60 2024-03-25T19:38:37.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.745+0000 7ff8e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:37.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.745+0000 7ff8e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:37.745 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.745+0000 7ff8e0ff9640 10 monclient: dump: 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.745+0000 7ff8e27fc640 4 set_mon_vals no callback set 2024-03-25T19:38:37.746 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.746+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7ff8d8019b80 con 0x7ff8e40b4d60 2024-03-25T19:38:37.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.746+0000 7ff8e3fff640 10 monclient: get_auth_request con 0x7ff8d006e160 auth_method 0 2024-03-25T19:38:37.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.747+0000 7ff8e0ff9640 10 client.4741.objecter ms_handle_connect 0x7ff8d006e160 2024-03-25T19:38:37.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.750+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7ff8d8053690 con 0x7ff8e40b4d60 2024-03-25T19:38:37.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.750+0000 7ff8e0ff9640 10 client.4741.objecter ms_dispatch 0x7ff8e4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:38:37.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.750+0000 7ff8e0ff9640 3 client.4741.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:38:37.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.750+0000 7ff8e0ff9640 3 client.4741.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:38:37.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.750+0000 7ff8e0ff9640 20 client.4741.objecter dump_active .. 0 homeless 2024-03-25T19:38:37.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.750+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8d804f1a0 con 0x7ff8e40b4d60 2024-03-25T19:38:37.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.750+0000 7ff8e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:37.750 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.750+0000 7ff8e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:37.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.977+0000 7ff8ea674640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:37.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.977+0000 7ff8ea674640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:37.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.977+0000 7ff8ea674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-25T19:38:37.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:37.977+0000 7ff8ea674640 1 -- 172.21.15.52:0/2270572202 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8e41daad0 con 0x7ff8e40b4d60 2024-03-25T19:38:38.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.000+0000 7ff8e0ff9640 1 -- 172.21.15.52:0/2270572202 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7ff8d80510d0 con 0x7ff8e40b4d60 2024-03-25T19:38:38.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.000+0000 7ff8e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:38.000 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.000+0000 7ff8e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:38.000 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:38:38.000 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:38.002 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.002+0000 7ff8ea674640 10 librados: watch_flush enter 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.002+0000 7ff8ea674640 10 librados: watch_flush exit 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.002+0000 7ff8ea674640 20 client.4741.objecter shutdown clearing up homeless session... 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.002+0000 7ff8ea674640 10 client.4741.objecter successfully canceled tick 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.002+0000 7ff8ea674640 1 -- 172.21.15.52:0/2270572202 >> v1:172.21.15.52:6800/3967427147 conn(0x7ff8d006e160 legacy=0x7ff8d0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.002+0000 7ff8ea674640 10 monclient: shutdown 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.002+0000 7ff8ea674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.002+0000 7ff8ea674640 1 -- 172.21.15.52:0/2270572202 >> v1:172.21.15.179:6789/0 conn(0x7ff8e40b4d60 legacy=0x7ff8e4132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.002+0000 7ff8ea674640 1 -- 172.21.15.52:0/2270572202 shutdown_connections 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.002+0000 7ff8ea674640 1 -- 172.21.15.52:0/2270572202 >> 172.21.15.52:0/2270572202 conn(0x7ff8e40617a0 msgr2=0x7ff8e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.003+0000 7ff8ea674640 1 -- 172.21.15.52:0/2270572202 shutdown_connections 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.003+0000 7ff8ea674640 1 -- 172.21.15.52:0/2270572202 wait complete. 2024-03-25T19:38:38.003 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:38.003+0000 7ff8ea674640 1 librados: shutdown 2024-03-25T19:38:38.014 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:38.272 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:31.871 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.320 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:31.919 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.360 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:31.959 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.374 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:31.916 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.382 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:31.924 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.396 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:31.995 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.405 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:31.947 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.408 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:32.007 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:38.444 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:32.043 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:44.821 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:44.936 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.936+0000 7f92c9aa8640 10 monclient: get_monmap_and_config 2024-03-25T19:38:44.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.936+0000 7f92c9aa8640 10 monclient: build_initial_monmap 2024-03-25T19:38:44.937 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.936+0000 7f92c9aa8640 10 monclient: monmap: 2024-03-25T19:38:44.937 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:44.937 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:44.937 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:38:44.938121+0000 2024-03-25T19:38:44.937 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:38:44.938121+0000 2024-03-25T19:38:44.937 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:44.937 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:44.937 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:44.937 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:38:44.937 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:38:44.938 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.937+0000 7f92c9aa8640 1 Processor -- start 2024-03-25T19:38:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.937+0000 7f92c9aa8640 1 -- start start 2024-03-25T19:38:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.937+0000 7f92c9aa8640 10 monclient: init 2024-03-25T19:38:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.938+0000 7f92c9aa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.938+0000 7f92c9aa8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.938+0000 7f92c9aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f92c40b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.938+0000 7f92c9aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f92c40b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.938+0000 7f92c9aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f92c414be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:44.939 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.938+0000 7f92c9aa8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c40af940 con 0x7f92c414be90 2024-03-25T19:38:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.938+0000 7f92c9aa8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c40b0a70 con 0x7f92c40b4d60 2024-03-25T19:38:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.938+0000 7f92c9aa8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c40b0d50 con 0x7f92c40b1730 2024-03-25T19:38:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.938+0000 7f92c9aa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:44.940 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.938+0000 7f92c9aa8640 10 monclient(hunting): authenticate will time out at 6573.119219s 2024-03-25T19:38:44.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.943+0000 7f92c2ffd640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f92c40b1730 0x7f92c40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:55690/0 (socket says 172.21.15.52:55690) 2024-03-25T19:38:44.943 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.943+0000 7f92c2ffd640 1 -- 172.21.15.52:0/852450894 learned_addr learned my addr 172.21.15.52:0/852450894 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:44.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.958+0000 7f92c27fc640 1 -- 172.21.15.52:0/852450894 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768081493 0 0) 0x7f92c40b0d50 con 0x7f92c40b1730 2024-03-25T19:38:44.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.958+0000 7f92c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:44.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.958+0000 7f92c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:44.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.958+0000 7f92c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:44.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.958+0000 7f92c27fc640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:38:44.958 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.958+0000 7f92c27fc640 1 -- 172.21.15.52:0/852450894 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac003520 con 0x7f92c40b1730 2024-03-25T19:38:44.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.961+0000 7f92c27fc640 1 -- 172.21.15.52:0/852450894 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183650392 0 0) 0x7f92c40b0a70 con 0x7f92c40b4d60 2024-03-25T19:38:44.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.961+0000 7f92c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:38:44.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.961+0000 7f92c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:44.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.961+0000 7f92c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:44.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.961+0000 7f92c27fc640 10 monclient(hunting): my global_id is 4744 2024-03-25T19:38:44.962 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.961+0000 7f92c27fc640 1 -- 172.21.15.52:0/852450894 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c40b0d50 con 0x7f92c40b4d60 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 1 -- 172.21.15.52:0/852450894 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1508474180 0 0) 0x7f92ac003520 con 0x7f92c40b1730 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 1 -- 172.21.15.52:0/852450894 >> v1:172.21.15.179:6789/0 conn(0x7f92c40b4d60 legacy=0x7f92c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 1 -- 172.21.15.52:0/852450894 >> v1:172.21.15.52:6789/0 conn(0x7f92c414be90 legacy=0x7f92c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 1 monclient: found mon.noname-b 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.52:6790/0 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 1 -- 172.21.15.52:0/852450894 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c40b1030 con 0x7f92c40b1730 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 10 monclient: _finish_auth 0 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 1 -- 172.21.15.52:0/852450894 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f92b8001690 con 0x7f92c40b1730 2024-03-25T19:38:44.965 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c27fc640 10 monclient: dump: 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c9aa8640 5 monclient: authenticate success, global_id 4724 2024-03-25T19:38:44.966 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.964+0000 7f92c9aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:44.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.971+0000 7f92c27fc640 1 -- 172.21.15.52:0/852450894 <== mon.2 v1:172.21.15.52:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92b80017f0 con 0x7f92c40b1730 2024-03-25T19:38:44.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.971+0000 7f92c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:44.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.971+0000 7f92c27fc640 1 -- 172.21.15.52:0/852450894 <== mon.2 v1:172.21.15.52:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f92b8001c00 con 0x7f92c40b1730 2024-03-25T19:38:44.971 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.971+0000 7f92c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.971+0000 7f92c27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.971+0000 7f92c27fc640 10 monclient: dump: 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.971+0000 7f92c9aa8640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:44.972 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.971+0000 7f92c9aa8640 4 set_mon_vals no callback set 2024-03-25T19:38:44.974 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.972+0000 7f92c9aa8640 10 monclient: shutdown 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.972+0000 7f92c9aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.972+0000 7f92c9aa8640 1 -- 172.21.15.52:0/852450894 >> v1:172.21.15.52:6790/0 conn(0x7f92c40b1730 legacy=0x7f92c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.972+0000 7f92c9aa8640 1 -- 172.21.15.52:0/852450894 shutdown_connections 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.972+0000 7f92c9aa8640 1 -- 172.21.15.52:0/852450894 >> 172.21.15.52:0/852450894 conn(0x7f92c40617a0 msgr2=0x7f92c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.972+0000 7f92c9aa8640 1 -- 172.21.15.52:0/852450894 shutdown_connections 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.972+0000 7f92c9aa8640 1 -- 172.21.15.52:0/852450894 wait complete. 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.972+0000 7f92c9aa8640 10 monclient: build_initial_monmap 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.972+0000 7f92c9aa8640 10 monclient: monmap: 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:38:44.975 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.973+0000 7f92c9aa8640 1 librados: starting msgr at 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.973+0000 7f92c9aa8640 1 librados: starting objecter 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.973+0000 7f92c9aa8640 1 Processor -- start 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.973+0000 7f92c9aa8640 1 -- start start 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.973+0000 7f92c9aa8640 1 librados: setting wanted keys 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.973+0000 7f92c9aa8640 1 librados: calling monclient init 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.973+0000 7f92c9aa8640 10 monclient: init 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.974+0000 7f92c9aa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.974+0000 7f92c9aa8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:44.976 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.974+0000 7f92c9aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f92c40b1730 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:44.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.974+0000 7f92c9aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f92c40b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:44.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.974+0000 7f92c9aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f92c414be90 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:44.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.974+0000 7f92c9aa8640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c4142080 con 0x7f92c40b1730 2024-03-25T19:38:44.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.974+0000 7f92c9aa8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c4138ab0 con 0x7f92c40b4d60 2024-03-25T19:38:44.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.974+0000 7f92c9aa8640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c41387b0 con 0x7f92c414be90 2024-03-25T19:38:44.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.974+0000 7f92c9aa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:44.977 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.974+0000 7f92c9aa8640 10 monclient(hunting): authenticate will time out at 6573.155079s 2024-03-25T19:38:44.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.990+0000 7f92c3fff640 1 --1- >> v1:172.21.15.52:6790/0 conn(0x7f92c414be90 0x7f92c41e32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.52:6790/0 says I am v1:172.21.15.52:55722/0 (socket says 172.21.15.52:55722) 2024-03-25T19:38:44.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.990+0000 7f92c3fff640 1 -- 172.21.15.52:0/3946730503 learned_addr learned my addr 172.21.15.52:0/3946730503 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:44.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.990+0000 7f92abfff640 10 client.?.objecter ms_handle_connect 0x7f92c414be90 2024-03-25T19:38:44.990 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.990+0000 7f92abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:44.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.994+0000 7f92abfff640 10 client.?.objecter ms_handle_connect 0x7f92c40b4d60 2024-03-25T19:38:44.994 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.994+0000 7f92abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:44.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.996+0000 7f92abfff640 10 client.?.objecter ms_handle_connect 0x7f92c40b1730 2024-03-25T19:38:44.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:44.996+0000 7f92abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:45.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.012+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576216467 0 0) 0x7f92c4138ab0 con 0x7f92c40b4d60 2024-03-25T19:38:45.012 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.012+0000 7f92abfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.012+0000 7f92abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.012+0000 7f92abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.012+0000 7f92abfff640 10 monclient(hunting): my global_id is 4747 2024-03-25T19:38:45.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.012+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b0004eb0 con 0x7f92c40b4d60 2024-03-25T19:38:45.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.013+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706512730 0 0) 0x7f92c41387b0 con 0x7f92c414be90 2024-03-25T19:38:45.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.013+0000 7f92abfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.013+0000 7f92abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.013+0000 7f92abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.013 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.013+0000 7f92abfff640 10 monclient(hunting): my global_id is 4727 2024-03-25T19:38:45.014 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.013+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c4138ab0 con 0x7f92c414be90 2024-03-25T19:38:45.019 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.019+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768934546 0 0) 0x7f92c4142080 con 0x7f92c40b1730 2024-03-25T19:38:45.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.019+0000 7f92abfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:45.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.019+0000 7f92abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:45.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.019+0000 7f92abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:45.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.019+0000 7f92abfff640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:38:45.020 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.019+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92c41387b0 con 0x7f92c40b1730 2024-03-25T19:38:45.025 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.025+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1544291246 0 0) 0x7f92b0004eb0 con 0x7f92c40b4d60 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.025+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c4142080 con 0x7f92c40b4d60 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.025+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f92b8001b00 con 0x7f92c40b4d60 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.025+0000 7f92abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.025+0000 7f92abfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.025+0000 7f92abfff640 10 monclient(hunting): dump: 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:45.026 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:45.027 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:45.027 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.025+0000 7f92abfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:45.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.036+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.2 v1:172.21.15.52:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2637138910 0 0) 0x7f92c4138ab0 con 0x7f92c414be90 2024-03-25T19:38:45.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.036+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 --> v1:172.21.15.52:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92b0004eb0 con 0x7f92c414be90 2024-03-25T19:38:45.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.036+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.2 v1:172.21.15.52:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f92bc001c10 con 0x7f92c414be90 2024-03-25T19:38:45.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.036+0000 7f92abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.036+0000 7f92abfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:45.036 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.036+0000 7f92abfff640 10 monclient(hunting): dump: 2024-03-25T19:38:45.036 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:45.036 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:45.036 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:45.037 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:45.037 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.037 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:45.037 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:45.037 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:45.037 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:45.037 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:45.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.047+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2495943660 0 0) 0x7f92c41387b0 con 0x7f92c40b1730 2024-03-25T19:38:45.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92c4138ab0 con 0x7f92c40b1730 2024-03-25T19:38:45.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f92b4001930 con 0x7f92c40b1730 2024-03-25T19:38:45.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:45.048 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 10 monclient(hunting): dump: 2024-03-25T19:38:45.048 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:45.048 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:45.048 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:45.048 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:45.049 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.049 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:45.049 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:45.049 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:45.049 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:45.049 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:45.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.2 v1:172.21.15.52:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2899973475 0 0) 0x7f92b0004eb0 con 0x7f92c414be90 2024-03-25T19:38:45.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 >> v1:172.21.15.179:6789/0 conn(0x7f92c40b4d60 legacy=0x7f92c41d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 >> v1:172.21.15.52:6789/0 conn(0x7f92c40b1730 legacy=0x7f92c4134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:45.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 1 monclient: found mon.c 2024-03-25T19:38:45.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:45.050 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 --> v1:172.21.15.52:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c41d54f0 con 0x7f92c414be90 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 10 monclient: _finish_auth 0 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92c9aa8640 5 monclient: authenticate success, global_id 4727 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92c9aa8640 10 monclient: _renew_subs 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92c9aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92c9aa8640 1 -- 172.21.15.52:0/3946730503 --> v1:172.21.15.52:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c4140bd0 con 0x7f92c414be90 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92c9aa8640 10 client.4727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92c9aa8640 10 monclient: _renew_subs 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92c9aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92c9aa8640 1 -- 172.21.15.52:0/3946730503 --> v1:172.21.15.52:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c41407d0 con 0x7f92c414be90 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.048+0000 7f92c9aa8640 1 librados: init done 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.049+0000 7f92c9aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.049+0000 7f92c9aa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:45.051 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.049+0000 7f92c9aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:38:45.052 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.049+0000 7f92c9aa8640 1 -- 172.21.15.52:0/3946730503 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c40b22c0 con 0x7f92c414be90 2024-03-25T19:38:45.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.371+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.2 v1:172.21.15.52:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92bc003970 con 0x7f92c414be90 2024-03-25T19:38:45.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.371+0000 7f92abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:45.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.371+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.2 v1:172.21.15.52:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f92bc002100 con 0x7f92c414be90 2024-03-25T19:38:45.372 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.371+0000 7f92abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.371+0000 7f92abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.371+0000 7f92abfff640 10 monclient: dump: 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.371+0000 7f92c1ffb640 4 set_mon_vals no callback set 2024-03-25T19:38:45.373 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.372+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.2 v1:172.21.15.52:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f92bc003d80 con 0x7f92c414be90 2024-03-25T19:38:45.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.373+0000 7f92c37fe640 10 monclient: get_auth_request con 0x7f92b006e110 auth_method 0 2024-03-25T19:38:45.374 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.373+0000 7f92abfff640 10 client.4727.objecter ms_handle_connect 0x7f92b006e110 2024-03-25T19:38:45.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.379+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.2 v1:172.21.15.52:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7f92bc054990 con 0x7f92c414be90 2024-03-25T19:38:45.379 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.379+0000 7f92abfff640 10 client.4727.objecter ms_dispatch 0x7f92c4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:38:45.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.379+0000 7f92abfff640 3 client.4727.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:38:45.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.379+0000 7f92abfff640 3 client.4727.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:38:45.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.379+0000 7f92abfff640 20 client.4727.objecter dump_active .. 0 homeless 2024-03-25T19:38:45.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.379+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.2 v1:172.21.15.52:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92bc0504a0 con 0x7f92c414be90 2024-03-25T19:38:45.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.379+0000 7f92abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:45.380 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.379+0000 7f92abfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:45.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.612+0000 7f92c9aa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:45.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.612+0000 7f92c9aa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:45.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.612+0000 7f92c9aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.52:6790/0 2024-03-25T19:38:45.612 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.612+0000 7f92c9aa8640 1 -- 172.21.15.52:0/3946730503 --> v1:172.21.15.52:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92c4140ed0 con 0x7f92c414be90 2024-03-25T19:38:45.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.658+0000 7f92abfff640 1 -- 172.21.15.52:0/3946730503 <== mon.2 v1:172.21.15.52:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7f92bc0523d0 con 0x7f92c414be90 2024-03-25T19:38:45.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.658+0000 7f92abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:45.659 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.658+0000 7f92abfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:45.659 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:38:45.659 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:45.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 10 librados: watch_flush enter 2024-03-25T19:38:45.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 10 librados: watch_flush exit 2024-03-25T19:38:45.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 20 client.4727.objecter shutdown clearing up homeless session... 2024-03-25T19:38:45.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 10 client.4727.objecter successfully canceled tick 2024-03-25T19:38:45.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 1 -- 172.21.15.52:0/3946730503 >> v1:172.21.15.52:6800/3967427147 conn(0x7f92b006e110 legacy=0x7f92b00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 10 monclient: shutdown 2024-03-25T19:38:45.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:45.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 1 -- 172.21.15.52:0/3946730503 >> v1:172.21.15.52:6790/0 conn(0x7f92c414be90 legacy=0x7f92c41e32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:45.662 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 1 -- 172.21.15.52:0/3946730503 shutdown_connections 2024-03-25T19:38:45.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 1 -- 172.21.15.52:0/3946730503 >> 172.21.15.52:0/3946730503 conn(0x7f92c40617a0 msgr2=0x7f92c40b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:45.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 1 -- 172.21.15.52:0/3946730503 shutdown_connections 2024-03-25T19:38:45.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 1 -- 172.21.15.52:0/3946730503 wait complete. 2024-03-25T19:38:45.663 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:45.661+0000 7f92c9aa8640 1 librados: shutdown 2024-03-25T19:38:45.673 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:52.480 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:52.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.596+0000 7f2a968c5640 10 monclient: get_monmap_and_config 2024-03-25T19:38:52.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.596+0000 7f2a968c5640 10 monclient: build_initial_monmap 2024-03-25T19:38:52.597 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.596+0000 7f2a968c5640 10 monclient: monmap: 2024-03-25T19:38:52.597 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:52.597 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:52.597 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:38:52.598389+0000 2024-03-25T19:38:52.597 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:38:52.598389+0000 2024-03-25T19:38:52.598 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:52.598 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:52.598 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:52.598 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-25T19:38:52.598 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-b 2024-03-25T19:38:52.598 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:52.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.597+0000 7f2a968c5640 1 Processor -- start 2024-03-25T19:38:52.599 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.597+0000 7f2a968c5640 1 -- start start 2024-03-25T19:38:52.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.597+0000 7f2a968c5640 10 monclient: init 2024-03-25T19:38:52.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.598+0000 7f2a968c5640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:52.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.598+0000 7f2a968c5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:52.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.598+0000 7f2a968c5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a900b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:52.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.598+0000 7f2a968c5640 10 monclient(hunting): picked mon.noname-b con 0x7f2a900b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:52.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.598+0000 7f2a968c5640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9014be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:52.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.598+0000 7f2a968c5640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a900af940 con 0x7f2a9014be90 2024-03-25T19:38:52.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.598+0000 7f2a968c5640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a900b0a70 con 0x7f2a900b4d60 2024-03-25T19:38:52.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.598+0000 7f2a968c5640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a900b0d50 con 0x7f2a900b1730 2024-03-25T19:38:52.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.598+0000 7f2a968c5640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:52.600 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.598+0000 7f2a968c5640 10 monclient(hunting): authenticate will time out at 6580.779482s 2024-03-25T19:38:52.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.608+0000 7f2a8ffff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f2a900b4d60 0x7f2a9013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:60672/0 (socket says 172.21.15.52:60672) 2024-03-25T19:38:52.609 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.608+0000 7f2a8ffff640 1 -- 172.21.15.52:0/4166120846 learned_addr learned my addr 172.21.15.52:0/4166120846 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:52.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.623+0000 7f2a8effd640 1 -- 172.21.15.52:0/4166120846 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247150982 0 0) 0x7f2a900b0a70 con 0x7f2a900b4d60 2024-03-25T19:38:52.623 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.623+0000 7f2a8effd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.623+0000 7f2a8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.623+0000 7f2a8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.623+0000 7f2a8effd640 10 monclient(hunting): my global_id is 4750 2024-03-25T19:38:52.624 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.623+0000 7f2a8effd640 1 -- 172.21.15.52:0/4166120846 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a78003520 con 0x7f2a900b4d60 2024-03-25T19:38:52.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.634+0000 7f2a8effd640 1 -- 172.21.15.52:0/4166120846 <== mon.2 v1:172.21.15.52:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769837922 0 0) 0x7f2a900b0d50 con 0x7f2a900b1730 2024-03-25T19:38:52.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.634+0000 7f2a8effd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.634+0000 7f2a8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.634+0000 7f2a8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.634+0000 7f2a8effd640 10 monclient(hunting): my global_id is 4730 2024-03-25T19:38:52.635 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.634+0000 7f2a8effd640 1 -- 172.21.15.52:0/4166120846 --> v1:172.21.15.52:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a900b0a70 con 0x7f2a900b1730 2024-03-25T19:38:52.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 1 -- 172.21.15.52:0/4166120846 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 636067420 0 0) 0x7f2a78003520 con 0x7f2a900b4d60 2024-03-25T19:38:52.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 1 -- 172.21.15.52:0/4166120846 >> v1:172.21.15.52:6790/0 conn(0x7f2a900b1730 legacy=0x7f2a900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 1 -- 172.21.15.52:0/4166120846 >> v1:172.21.15.52:6789/0 conn(0x7f2a9014be90 legacy=0x7f2a9014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:52.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 1 monclient: found mon.noname-c 2024-03-25T19:38:52.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:52.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-25T19:38:52.637 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 1 -- 172.21.15.52:0/4166120846 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a900b1030 con 0x7f2a900b4d60 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 10 monclient: _finish_auth 0 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 1 -- 172.21.15.52:0/4166120846 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2a80001690 con 0x7f2a900b4d60 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a968c5640 5 monclient: authenticate success, global_id 4750 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a968c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a8effd640 10 monclient: dump: 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.638 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:52.639 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:52.639 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:52.639 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:52.639 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:52.639 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.636+0000 7f2a968c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:52.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.643+0000 7f2a8effd640 1 -- 172.21.15.52:0/4166120846 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a800017f0 con 0x7f2a900b4d60 2024-03-25T19:38:52.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.643+0000 7f2a8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:52.643 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.643+0000 7f2a8effd640 1 -- 172.21.15.52:0/4166120846 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2a80001c00 con 0x7f2a900b4d60 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.643+0000 7f2a968c5640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.643+0000 7f2a968c5640 4 set_mon_vals no callback set 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.643+0000 7f2a8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.643+0000 7f2a8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.643+0000 7f2a8effd640 10 monclient: dump: 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:52.644 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:52.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.644+0000 7f2a968c5640 10 monclient: shutdown 2024-03-25T19:38:52.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.644+0000 7f2a968c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:52.646 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.644+0000 7f2a968c5640 1 -- 172.21.15.52:0/4166120846 >> v1:172.21.15.179:6789/0 conn(0x7f2a900b4d60 legacy=0x7f2a9013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.644+0000 7f2a968c5640 1 -- 172.21.15.52:0/4166120846 shutdown_connections 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.644+0000 7f2a968c5640 1 -- 172.21.15.52:0/4166120846 >> 172.21.15.52:0/4166120846 conn(0x7f2a900617a0 msgr2=0x7f2a90061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.644+0000 7f2a968c5640 1 -- 172.21.15.52:0/4166120846 shutdown_connections 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.644+0000 7f2a968c5640 1 -- 172.21.15.52:0/4166120846 wait complete. 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.644+0000 7f2a968c5640 10 monclient: build_initial_monmap 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.644+0000 7f2a968c5640 10 monclient: monmap: 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:epoch 0 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:last_changed 0.000000 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:created 0.000000 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 1 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.noname-a 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.noname-c 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:52.647 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.645+0000 7f2a968c5640 1 librados: starting msgr at 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.645+0000 7f2a968c5640 1 librados: starting objecter 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.645+0000 7f2a968c5640 1 Processor -- start 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.645+0000 7f2a968c5640 1 -- start start 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.645+0000 7f2a968c5640 1 librados: setting wanted keys 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.645+0000 7f2a968c5640 1 librados: calling monclient init 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.645+0000 7f2a968c5640 10 monclient: init 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.646+0000 7f2a968c5640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.646+0000 7f2a968c5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.646+0000 7f2a968c5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a900b1730 addr v1:172.21.15.52:6790/0 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.646+0000 7f2a968c5640 10 monclient(hunting): picked mon.noname-b con 0x7f2a900b4d60 addr v1:172.21.15.179:6789/0 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.646+0000 7f2a968c5640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9014be90 addr v1:172.21.15.52:6789/0 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.646+0000 7f2a968c5640 1 -- --> v1:172.21.15.52:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a901d7ae0 con 0x7f2a9014be90 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.646+0000 7f2a968c5640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a901426d0 con 0x7f2a900b4d60 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.646+0000 7f2a968c5640 1 -- --> v1:172.21.15.52:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a90142860 con 0x7f2a900b1730 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.646+0000 7f2a968c5640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:52.648 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.646+0000 7f2a968c5640 10 monclient(hunting): authenticate will time out at 6580.827137s 2024-03-25T19:38:52.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.652+0000 7f2a8f7fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f2a900b4d60 0x7f2a9013abd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.52:60698/0 (socket says 172.21.15.52:60698) 2024-03-25T19:38:52.652 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.652+0000 7f2a8f7fe640 1 -- 172.21.15.52:0/256191845 learned_addr learned my addr 172.21.15.52:0/256191845 (peer_addr_for_me v1:172.21.15.52:0/0) 2024-03-25T19:38:52.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.653+0000 7f2a8cff9640 10 client.?.objecter ms_handle_connect 0x7f2a900b4d60 2024-03-25T19:38:52.653 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.653+0000 7f2a8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:52.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.657+0000 7f2a8cff9640 10 client.?.objecter ms_handle_connect 0x7f2a9014be90 2024-03-25T19:38:52.658 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.657+0000 7f2a8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:52.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.665+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 <== mon.0 v1:172.21.15.52:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765775514 0 0) 0x7f2a901d7ae0 con 0x7f2a9014be90 2024-03-25T19:38:52.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.665+0000 7f2a8cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.665+0000 7f2a8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.665 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.665+0000 7f2a8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.665+0000 7f2a8cff9640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:38:52.666 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.665+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 --> v1:172.21.15.52:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a7c004eb0 con 0x7f2a9014be90 2024-03-25T19:38:52.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.670+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269078783 0 0) 0x7f2a901426d0 con 0x7f2a900b4d60 2024-03-25T19:38:52.670 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.670+0000 7f2a8cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:52.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.670+0000 7f2a8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:52.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.670+0000 7f2a8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:52.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.670+0000 7f2a8cff9640 10 monclient(hunting): my global_id is 4753 2024-03-25T19:38:52.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.670+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a901d7ae0 con 0x7f2a900b4d60 2024-03-25T19:38:52.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.671+0000 7f2a8cff9640 10 client.?.objecter ms_handle_connect 0x7f2a900b1730 2024-03-25T19:38:52.671 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.671+0000 7f2a8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:52.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.675+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 <== mon.0 v1:172.21.15.52:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2107793856 0 0) 0x7f2a7c004eb0 con 0x7f2a9014be90 2024-03-25T19:38:52.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.675+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 --> v1:172.21.15.52:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a901426d0 con 0x7f2a9014be90 2024-03-25T19:38:52.675 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.675+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 <== mon.0 v1:172.21.15.52:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2a88001b20 con 0x7f2a9014be90 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.675+0000 7f2a8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.675+0000 7f2a8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.675+0000 7f2a8cff9640 10 monclient(hunting): dump: 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:52.676 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.675+0000 7f2a8cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:52.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 <== mon.0 v1:172.21.15.52:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1446549310 0 0) 0x7f2a901426d0 con 0x7f2a9014be90 2024-03-25T19:38:52.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 >> v1:172.21.15.52:6790/0 conn(0x7f2a900b1730 legacy=0x7f2a901d8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 >> v1:172.21.15.179:6789/0 conn(0x7f2a900b4d60 legacy=0x7f2a9013abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:52.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a8cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:38:52.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a8cff9640 1 monclient: found mon.a 2024-03-25T19:38:52.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:52.682 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a8cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 --> v1:172.21.15.52:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a901d7690 con 0x7f2a9014be90 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a8cff9640 10 monclient: _finish_auth 0 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a8cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a968c5640 5 monclient: authenticate success, global_id 4737 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a968c5640 10 monclient: _renew_subs 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a968c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a968c5640 1 -- 172.21.15.52:0/256191845 --> v1:172.21.15.52:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a90141cd0 con 0x7f2a9014be90 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a968c5640 10 client.4737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a968c5640 10 monclient: _renew_subs 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a968c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a968c5640 1 -- 172.21.15.52:0/256191845 --> v1:172.21.15.52:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a90142c70 con 0x7f2a9014be90 2024-03-25T19:38:52.683 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.681+0000 7f2a968c5640 1 librados: init done 2024-03-25T19:38:52.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.682+0000 7f2a968c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:52.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.682+0000 7f2a968c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:52.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.682+0000 7f2a968c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:38:52.684 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.682+0000 7f2a968c5640 1 -- 172.21.15.52:0/256191845 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a900627b0 con 0x7f2a9014be90 2024-03-25T19:38:52.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.905+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 <== mon.0 v1:172.21.15.52:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a88002030 con 0x7f2a9014be90 2024-03-25T19:38:52.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.905+0000 7f2a8cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:52.905 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.905+0000 7f2a8e7fc640 4 set_mon_vals no callback set 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.946+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 <== mon.0 v1:172.21.15.52:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3731802535 0 0) 0x7f2a88003b10 con 0x7f2a9014be90 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.946+0000 7f2a8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.946+0000 7f2a8cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.946+0000 7f2a8cff9640 10 monclient: dump: 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi052.stderr:epoch 1 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi052.stderr:fsid ef6405ee-982f-417b-9446-354282e3761f 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi052.stderr:last_changed 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi052.stderr:created 2024-03-25T19:26:35.450503+0000 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi052.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:52.946 INFO:teuthology.misc.health.smithi052.stderr:election_strategy: 3 2024-03-25T19:38:52.947 INFO:teuthology.misc.health.smithi052.stderr:0: v1:172.21.15.52:6789/0 mon.a 2024-03-25T19:38:52.947 INFO:teuthology.misc.health.smithi052.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-25T19:38:52.947 INFO:teuthology.misc.health.smithi052.stderr:2: v1:172.21.15.52:6790/0 mon.c 2024-03-25T19:38:52.947 INFO:teuthology.misc.health.smithi052.stderr: 2024-03-25T19:38:52.996 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.996+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 <== mon.0 v1:172.21.15.52:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3021675231 0 0) 0x7f2a88003cf0 con 0x7f2a9014be90 2024-03-25T19:38:52.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.997+0000 7f2a8ffff640 10 monclient: get_auth_request con 0x7f2a7c06dcc0 auth_method 0 2024-03-25T19:38:52.997 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:52.997+0000 7f2a8cff9640 10 client.4737.objecter ms_handle_connect 0x7f2a7c06dcc0 2024-03-25T19:38:53.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.007+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 <== mon.0 v1:172.21.15.52:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3684630143 0 0) 0x7f2a880537f0 con 0x7f2a9014be90 2024-03-25T19:38:53.007 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.007+0000 7f2a8cff9640 10 client.4737.objecter ms_dispatch 0x7f2a90000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:38:53.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.007+0000 7f2a8cff9640 3 client.4737.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:38:53.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.007+0000 7f2a8cff9640 3 client.4737.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:38:53.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.007+0000 7f2a8cff9640 20 client.4737.objecter dump_active .. 0 homeless 2024-03-25T19:38:53.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.007+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 <== mon.0 v1:172.21.15.52:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a8804f130 con 0x7f2a9014be90 2024-03-25T19:38:53.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.007+0000 7f2a8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:53.008 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.007+0000 7f2a8cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:53.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.225+0000 7f2a968c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:53.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.225+0000 7f2a968c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:53.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.225+0000 7f2a968c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.52:6789/0 2024-03-25T19:38:53.226 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.225+0000 7f2a968c5640 1 -- 172.21.15.52:0/256191845 --> v1:172.21.15.52:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a90069f40 con 0x7f2a9014be90 2024-03-25T19:38:53.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.270+0000 7f2a8cff9640 1 -- 172.21.15.52:0/256191845 <== mon.0 v1:172.21.15.52:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 927332156) 0x7f2a88051230 con 0x7f2a9014be90 2024-03-25T19:38:53.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.270+0000 7f2a8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:53.271 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.270+0000 7f2a8cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:53.271 INFO:teuthology.misc.health.smithi052.stdout: 2024-03-25T19:38:53.271 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":11,"fsid":"ef6405ee-982f-417b-9446-354282e3761f","created":"2024-03-25T19:27:23.722145+0000","modified":"2024-03-25T19:37:27.972334+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:37:27.972334+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"037ba8f0-68fe-4829-a15d-6cfeb9255161","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1c191de0-2061-4c78-97e5-c60bc8ba9c84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"8a2511b0-7d13-4706-be6b-6a51ff228663","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0ac59362-fcca-46ec-8c8b-13d7e7829481","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2fa32263-00b3-4fe0-85db-58725da804fe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"25acb8c3-a73f-4fbb-8b96-840b236c0235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a846d437-978e-4458-a2dd-254e5eb4275c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0550ddb7-e774-4908-8042-591d2ea0875f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:53.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.272+0000 7f2a968c5640 10 librados: watch_flush enter 2024-03-25T19:38:53.273 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.272+0000 7f2a968c5640 10 librados: watch_flush exit 2024-03-25T19:38:53.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.272+0000 7f2a968c5640 20 client.4737.objecter shutdown clearing up homeless session... 2024-03-25T19:38:53.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.272+0000 7f2a968c5640 10 client.4737.objecter successfully canceled tick 2024-03-25T19:38:53.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.272+0000 7f2a968c5640 1 -- 172.21.15.52:0/256191845 >> v1:172.21.15.52:6800/3967427147 conn(0x7f2a7c06dcc0 legacy=0x7f2a7c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.273+0000 7f2a968c5640 10 monclient: shutdown 2024-03-25T19:38:53.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.273+0000 7f2a968c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:53.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.273+0000 7f2a968c5640 1 -- 172.21.15.52:0/256191845 >> v1:172.21.15.52:6789/0 conn(0x7f2a9014be90 legacy=0x7f2a901415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:53.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.273+0000 7f2a968c5640 1 -- 172.21.15.52:0/256191845 shutdown_connections 2024-03-25T19:38:53.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.273+0000 7f2a968c5640 1 -- 172.21.15.52:0/256191845 >> 172.21.15.52:0/256191845 conn(0x7f2a900617a0 msgr2=0x7f2a900b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:53.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.273+0000 7f2a968c5640 1 -- 172.21.15.52:0/256191845 shutdown_connections 2024-03-25T19:38:53.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.273+0000 7f2a968c5640 1 -- 172.21.15.52:0/256191845 wait complete. 2024-03-25T19:38:53.274 INFO:teuthology.misc.health.smithi052.stderr:2024-03-25T19:38:53.273+0000 7f2a968c5640 1 librados: shutdown 2024-03-25T19:38:53.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:53.284 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:38:53.285 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:38:53.285 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:38:53.285 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:38:55.181 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:11:25.844 35419== 2024-03-25T19:38:55.181 INFO:tasks.ceph.osd.0.smithi052.stderr:==00:00:11:25.844 35419== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:38:55.223 INFO:tasks.ceph.osd.0.smithi052.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:38:58.312 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:51.911 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.355 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:51.897 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.357 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:51.899 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.366 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:51.965 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.389 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:11:51.988 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.396 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:11:51.938 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.425 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:52.024 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.431 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:52.030 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:58.461 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:11:52.060 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.387 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:38:59.387 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:38:59.388 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:38:59.388 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:38:59.388 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:01.271 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:11:31.950 35418== 2024-03-25T19:39:01.271 INFO:tasks.ceph.osd.1.smithi052.stderr:==00:00:11:31.950 35418== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:01.308 INFO:tasks.ceph.osd.1.smithi052.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:39:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:39:05.490 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:39:05.491 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:39:05.491 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:39:05.491 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:07.459 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:11:38.113 35421== 2024-03-25T19:39:07.459 INFO:tasks.ceph.osd.2.smithi052.stderr:==00:00:11:38.113 35421== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:07.500 INFO:tasks.ceph.osd.2.smithi052.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:39:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:39:11.593 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:39:11.593 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:39:11.593 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:39:11.594 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:13.380 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:11:44.040 35420== 2024-03-25T19:39:13.381 INFO:tasks.ceph.osd.3.smithi052.stderr:==00:00:11:44.040 35420== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:13.422 INFO:tasks.ceph.osd.3.smithi052.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:39:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:39:17.695 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:39:17.696 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:39:17.696 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:39:17.696 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:18.399 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:12:11.941 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.401 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:12:11.942 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.402 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:12:12.000 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.405 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:12:12.004 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.421 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:12:11.963 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.425 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:12:12.024 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.457 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:12:12.056 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.462 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:12:12.061 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:18.483 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:12:12.082 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.557 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:11:50.233 35758== 2024-03-25T19:39:19.557 INFO:tasks.ceph.osd.4.smithi179.stderr:==00:00:11:50.233 35758== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:19.594 INFO:tasks.ceph.osd.4.smithi179.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:39:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:39:23.798 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi179 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:39:23.799 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:39:23.799 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:39:23.799 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:25.618 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:11:56.290 35760== 2024-03-25T19:39:25.618 INFO:tasks.ceph.osd.5.smithi179.stderr:==00:00:11:56.290 35760== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:25.651 INFO:tasks.ceph.osd.5.smithi179.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:39:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:39:29.901 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi179 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:39:29.901 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:39:29.902 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:39:29.902 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:31.614 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:12:02.286 35759== 2024-03-25T19:39:31.614 INFO:tasks.ceph.osd.6.smithi179.stderr:==00:00:12:02.286 35759== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:31.662 INFO:tasks.ceph.osd.6.smithi179.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:39:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:39:36.003 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi179 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:39:36.004 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:39:36.004 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:39:36.004 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:37.436 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:12:08.085 35761== 2024-03-25T19:39:37.436 INFO:tasks.ceph.osd.7.smithi179.stderr:==00:00:12:08.085 35761== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:37.465 INFO:tasks.ceph.osd.7.smithi179.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:39:38.456 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:12:32.054 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.460 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:12:32.059 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.471 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:12:32.070 35103== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.501 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:12:32.043 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.502 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:12:32.044 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.503 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:12:32.103 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.506 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:12:32.105 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.513 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:12:32.055 35215== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:38.517 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:12:32.117 35105== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:39:42.106 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi179 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:39:42.106 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:39:42.106 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:39:42.107 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:39:42.107 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:42.108 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:39:42.108+0000 7f7f92829640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35112) UID: 0 2024-03-25T19:39:42.108 INFO:tasks.ceph.mgr.y.smithi052.stderr:2024-03-25T19:39:42.108+0000 7f7f92829640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T19:39:42.135 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:39:42.135 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:39:42.135 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:42.162 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:39:42.162 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:39:42.162 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:39:42.162 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:42.191 INFO:tasks.ceph.mon.a.smithi052.stderr:2024-03-25T19:39:42.180+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35072) UID: 0 2024-03-25T19:39:42.191 INFO:tasks.ceph.mon.a.smithi052.stderr:2024-03-25T19:39:42.182+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:39:42.802 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:12:36.401 35103== 2024-03-25T19:39:42.802 INFO:tasks.ceph.mon.a.smithi052.stderr:==00:00:12:36.401 35103== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:42.812 INFO:tasks.ceph.mon.a.smithi052.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:39:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:39:48.264 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:39:48.264 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:39:48.264 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:39:48.264 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:48.282 INFO:tasks.ceph.mon.c.smithi052.stderr:2024-03-25T19:39:48.270+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35074) UID: 0 2024-03-25T19:39:48.283 INFO:tasks.ceph.mon.c.smithi052.stderr:2024-03-25T19:39:48.271+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:39:48.704 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:12:42.303 35105== 2024-03-25T19:39:48.704 INFO:tasks.ceph.mon.c.smithi052.stderr:==00:00:12:42.303 35105== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:48.713 INFO:tasks.ceph.mon.c.smithi052.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:39:54.366 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:39:54.366 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:39:54.367 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:39:54.367 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:39:54.367 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:39:54.383 INFO:tasks.ceph.mon.b.smithi179.stderr:2024-03-25T19:39:54.372+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35211) UID: 0 2024-03-25T19:39:54.383 INFO:tasks.ceph.mon.b.smithi179.stderr:2024-03-25T19:39:54.374+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T19:39:54.879 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:12:48.421 35215== 2024-03-25T19:39:54.879 INFO:tasks.ceph.mon.b.smithi179.stderr:==00:00:12:48.421 35215== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:39:54.888 INFO:tasks.ceph.mon.b.smithi179.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:40:00.469 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:40:00.469 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi179 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:40:00.470 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:40:00.470 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:40:00.470 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:40:00.520 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-25T19:40:00.520 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:00.666 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-25T19:40:00.666 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:00.734 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-25T19:40:00.734 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:00.798 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi052.front.sepia.ceph.com 2024-03-25T19:40:00.799 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:00.915 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-25T19:40:00.915 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:01.054 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-25T19:40:01.055 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:01.151 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-25T19:40:01.151 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:01.234 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-25T19:40:01.234 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:01.291 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:40:01.293 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315/data/mon.a.tgz 2024-03-25T19:40:01.293 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-03-25T19:40:01.311 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.N0tdM0gICO 2024-03-25T19:40:01.311 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.N0tdM0gICO 2024-03-25T19:40:01.425 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.N0tdM0gICO 2024-03-25T19:40:01.513 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.N0tdM0gICO is 133KB 2024-03-25T19:40:01.565 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.N0tdM0gICO 2024-03-25T19:40:01.580 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315/data/mon.c.tgz 2024-03-25T19:40:01.580 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-03-25T19:40:01.635 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.WT3ykTNN37 2024-03-25T19:40:01.635 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.WT3ykTNN37 2024-03-25T19:40:01.724 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.WT3ykTNN37 2024-03-25T19:40:01.809 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.WT3ykTNN37 is 133KB 2024-03-25T19:40:01.860 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.WT3ykTNN37 2024-03-25T19:40:01.873 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315/data/mon.b.tgz 2024-03-25T19:40:01.873 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-03-25T19:40:01.892 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.LmaMS0CbIw 2024-03-25T19:40:01.892 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.LmaMS0CbIw 2024-03-25T19:40:02.001 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.LmaMS0CbIw 2024-03-25T19:40:02.090 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.LmaMS0CbIw is 133KB 2024-03-25T19:40:02.142 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.LmaMS0CbIw 2024-03-25T19:40:02.157 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:40:02.157 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:40:02.160 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:40:02.543 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:40:02.792 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:40:02.794 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315/remote/smithi052/crash 2024-03-25T19:40:02.795 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:40:02.824 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315/remote/smithi179/crash 2024-03-25T19:40:02.825 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:40:02.853 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:40:02.853 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:40:02.868 DEBUG:teuthology.orchestra.run.smithi179:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:40:02.894 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41888.log 2024-03-25T19:40:02.947 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:40:02.947 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35488.log 2024-03-25T19:40:02.947 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:40:02.947 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.35488.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:40:02.947 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:40:02.947 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:40:02.947 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:40:02.947 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/valgrind/osd.5.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/valgrind/osd.6.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35488.log.gz 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35368.log 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/valgrind/osd.4.log: 72.1% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.35368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35408.log 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- 86.4% /var/log/ceph/ceph-client.admin.35608.log -- replaced with /var/log/ceph/ceph-client.admin.35368.log.gz 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.35408.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:40:02.948 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.35608.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35408.log.gz 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35608.log.gz 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose --/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.35528.log 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35659.log 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.35528.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.35659.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35528.log.gz 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr: 86.8%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.35659.log.gz 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35568.log 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35448.log 2024-03-25T19:40:02.949 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.35568.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35568.log.gz 2024-03-25T19:40:02.950 INFO:teuthology.orchestra.run.smithi179.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:40:02.950 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.35448.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35448.log.gz 2024-03-25T19:40:02.950 INFO:teuthology.orchestra.run.smithi179.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:40:02.952 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41458.log 2024-03-25T19:40:02.952 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42317.log 2024-03-25T19:40:02.952 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41888.log: /var/log/ceph/ceph-client.admin.41458.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:40:02.952 INFO:teuthology.orchestra.run.smithi052.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41888.log.gz 2024-03-25T19:40:02.952 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.42317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41732.log 2024-03-25T19:40:02.952 INFO:teuthology.orchestra.run.smithi052.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41458.log.gz 2024-03-25T19:40:02.952 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42239.log 2024-03-25T19:40:02.952 INFO:teuthology.orchestra.run.smithi052.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42317.log.gz 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41849.log 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.42239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41614.log 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41732.log.gz 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40950.log 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42239.log.gz 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41614.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41849.log.gz 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40248.log 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39897.log 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41614.log.gz 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40248.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40950.log.gzgzip 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr: -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40248.log.gz 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:40:02.953 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39897.log: /var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39897.log.gz 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41380.log 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41145.log 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41380.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:40:02.954 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41692.log 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39531.log 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41380.log.gz 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41692.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41145.log.gz 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35519.log 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41106.log 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41692.log.gz 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.35519.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39531.log.gz 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41028.log 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39687.log 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35519.log.gz 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40170.log 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41106.log.gz 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39687.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41028.log.gz 2024-03-25T19:40:02.955 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39219.log 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38818.log 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39453.log 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40170.log.gz 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39219.log.gz 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.38818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42278.log 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40053.log 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38818.log.gz 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.42278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41653.log 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39453.log.gz 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40053.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42278.log.gz 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41340.log 2024-03-25T19:40:02.956 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41966.log 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40053.log.gz 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35417.log 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41653.log.gz 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5%/var/log/ceph/ceph-client.admin.41966.log: -- replaced with /var/log/ceph/ceph-client.admin.41340.log.gz 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.35417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42122.log 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41966.log.gz 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35114.log 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35417.log.gz 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.42122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40404.log 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.35114.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42122.log.gz 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42161.log 2024-03-25T19:40:02.957 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39936.log 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35114.log.gz 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.42161.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40404.log.gz 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41575.log 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr:gzip 86.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.42161.log.gz --verbose 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr: -- /var/log/ceph/ceph-client.admin.42083.log 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39936.log: /var/log/ceph/ceph-client.admin.41575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39609.log 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.42083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39648.log 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39936.log.gz 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41575.log.gz 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41419.log 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42083.log.gz 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35654.log 2024-03-25T19:40:02.958 INFO:teuthology.orchestra.run.smithi052.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39609.log.gz 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41419.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39648.log.gz 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.35654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40482.log 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41419.log.gz 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.1.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35654.log.gz 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41184.log 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39061.log 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39180.log 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40482.log.gz 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41184.log.gz 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40131.log 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41771.log 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40131.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39061.log.gz 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr: 87.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.39180.log.gz -5 2024-03-25T19:40:02.959 INFO:teuthology.orchestra.run.smithi052.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42200.log 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40131.log.gz 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40326.log 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.42200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40794.log 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41771.log.gz 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40326.log: gzip -5 --verbose 86.5% -- -- replaced with /var/log/ceph/ceph-client.admin.42200.log.gz /var/log/ceph/ceph-client.admin.40716.log 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40794.log: gzip 86.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.40326.log.gz --verbose 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr: -- /var/log/ceph/ceph-client.admin.40755.log 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42044.log 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40794.log.gz 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40989.log 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40716.log.gz 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40755.log.gz 2024-03-25T19:40:02.960 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.42044.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40872.log 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42044.log.gz 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40833.log 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40989.log.gz 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40365.log 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39818.log 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40872.log.gz 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40365.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40833.log.gz 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42005.log 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36533.log 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39818.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40365.log.gz 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.42005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40911.log 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39818.log.gz 2024-03-25T19:40:02.961 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.36533.log: 86.7%gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29262.log 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42005.log.gz 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41262.log 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36533.log.gz 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph.tmp-client.admin.29262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39570.log 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29262.log.gz 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40911.log.gz 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40014.log 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39570.log: gzip -5 --verbose -- 86.3% /var/log/ceph/ceph-client.admin.39778.log 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41262.log.gz 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40014.log: gzip -5 --verbose -- 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39570.log.gz 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr: /var/log/ceph/ceph-client.admin.39858.log 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40014.log.gz 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40287.log 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39022.log 2024-03-25T19:40:02.962 INFO:teuthology.orchestra.run.smithi052.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39778.log.gz 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40638.log 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39858.log.gz 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41536.log 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40287.log.gz 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40443.log 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39022.log.gz 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41536.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40638.log.gz 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40443.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41536.log.gz 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39336.log 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5/var/log/ceph/ceph-mgr.y.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40443.log.gz 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41223.log 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39141.log 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39336.log.gz 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40677.log 2024-03-25T19:40:02.963 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5/var/log/ceph/ceph-client.admin.39141.log: --verbose 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41223.log.gz 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr: -- /var/log/ceph/ceph-client.admin.40599.log 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40677.log: gzip 86.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.39297.log 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39141.log.gz 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40677.log.gz 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39258.log 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41810.log 87.1% 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40599.log.gz 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41067.log 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39297.log.gz 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41810.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39258.log.gz 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39414.log 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41810.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40521.log 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr: 87.2%/var/log/ceph/ceph-osd.0.log: -- replaced with /var/log/ceph/ceph-client.admin.41067.log.gz 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40209.log 2024-03-25T19:40:02.964 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40521.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39414.log.gzgzip 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39975.log 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40521.log.gz 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40209.log: /var/log/ceph/ceph.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40209.log.gz 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41927.log 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr: 92.9% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-osd.2.log: 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39975.log.gz 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39726.log 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39492.log 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr: 86.1%/var/log/ceph/ceph-client.admin.39726.log: -- replaced with /var/log/ceph/ceph-client.admin.41927.log.gz 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39726.log.gz 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40092.log 2024-03-25T19:40:02.965 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41497.log 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.40092.log: /var/log/ceph/ceph-client.admin.39492.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40092.log.gz 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39492.log.gz 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40560.log 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41301.log 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41497.log: /var/log/ceph/ceph-client.admin.40560.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41497.log.gz 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40560.log.gz 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39102.log 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.41301.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41301.log.gz 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39375.log 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37556.log 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.39102.log: /var/log/ceph/ceph-client.admin.39375.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39102.log.gz 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39375.log.gz 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:40:02.966 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.37556.log: /var/log/ceph/ceph-mon.c.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37556.log.gz 2024-03-25T19:40:03.002 INFO:teuthology.orchestra.run.smithi052.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:40:03.758 INFO:teuthology.orchestra.run.smithi052.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:40:03.807 INFO:teuthology.orchestra.run.smithi179.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:40:03.822 INFO:teuthology.orchestra.run.smithi179.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:40:03.844 INFO:teuthology.orchestra.run.smithi179.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:40:04.032 INFO:teuthology.orchestra.run.smithi052.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:40:04.086 INFO:teuthology.orchestra.run.smithi179.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:40:04.115 INFO:teuthology.orchestra.run.smithi052.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:40:04.139 INFO:teuthology.orchestra.run.smithi052.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:40:08.020 INFO:teuthology.orchestra.run.smithi052.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:40:08.174 INFO:teuthology.orchestra.run.smithi052.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:40:08.176 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-03-25T19:40:08.176 INFO:teuthology.orchestra.run.smithi052.stderr:real 0m5.295s 2024-03-25T19:40:08.176 INFO:teuthology.orchestra.run.smithi052.stderr:user 0m14.429s 2024-03-25T19:40:08.177 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m0.504s 2024-03-25T19:40:08.219 INFO:teuthology.orchestra.run.smithi179.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:40:08.221 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-25T19:40:08.222 INFO:teuthology.orchestra.run.smithi179.stderr:real 0m5.306s 2024-03-25T19:40:08.222 INFO:teuthology.orchestra.run.smithi179.stderr:user 0m8.858s 2024-03-25T19:40:08.222 INFO:teuthology.orchestra.run.smithi179.stderr:sys 0m0.302s 2024-03-25T19:40:08.222 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:40:08.223 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315/remote/smithi052/log 2024-03-25T19:40:08.224 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:40:10.630 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315/remote/smithi179/log 2024-03-25T19:40:10.631 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:40:12.162 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:40:12.364 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1462e99bbde64254ac7447b7d77a04bd Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:40:12.368 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:40:12.378 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:40:12.392 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:40:12.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:40:12.406 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:40:12.479 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:40:12.487 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:40:12.488 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:40:12.490 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:40:12.504 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:40:12.506 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:40:12.509 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:40:12.510 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-03-25T19:40:12.510 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 -252us[ -252us] +/- 67ms 2024-03-25T19:40:12.510 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -301us[ -345us] +/- 53ms 2024-03-25T19:40:12.510 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +789us[ +789us] +/- 88ms 2024-03-25T19:40:12.510 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:40:12.512 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:40:12.512 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-03-25T19:40:12.512 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 -243us[ -243us] +/- 67ms 2024-03-25T19:40:12.512 INFO:teuthology.orchestra.run.smithi179.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -295us[ -314us] +/- 53ms 2024-03-25T19:40:12.512 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 +791us[ +771us] +/- 88ms 2024-03-25T19:40:12.513 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:40:12.513 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:40:12.525 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:40:12.526 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:40:12.538 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:40:12.625 DEBUG:teuthology.orchestra.run.smithi179:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:40:12.698 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:40:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:40:12.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:40:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:40:12.762 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:40:12.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:40:12.782 INFO:teuthology.task.internal:Duration was 6334.770914 seconds 2024-03-25T19:40:12.782 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:40:12.793 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:40:12.793 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:40:12.796 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:40:12.838 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:40:12.847 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:40:13.210 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:40:13.210 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2024-03-25T19:40:13.211 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:40:13.232 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2024-03-25T19:40:13.232 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:40:13.257 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:40:13.258 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:40:13.274 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:40:13.326 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:40:13.326 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:40:13.344 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:40:13.468 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:40:13.477 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:40:13.478 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:40:13.504 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:40:13.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:40:13.540 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:40:13.547 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:40:13.570 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2024-03-25T19:40:13.596 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2024-03-25T19:40:13.620 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:40:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:40:13.647 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:40:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:40:13.675 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:40:13.684 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:40:13.685 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315/remote/smithi052 2024-03-25T19:40:13.685 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:40:13.721 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315/remote/smithi179 2024-03-25T19:40:13.721 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:40:13.756 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:40:13.756 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:40:13.758 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:40:13.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:40:13.817 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:40:13.817 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:40:13.830 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:40:13.830 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:40:13.832 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:40:13.844 INFO:teuthology.orchestra.run.smithi052.stdout: 265832 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:40 /home/ubuntu/cephtest 2024-03-25T19:40:13.860 INFO:teuthology.orchestra.run.smithi179.stdout: 266042 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:40 /home/ubuntu/cephtest 2024-03-25T19:40:13.862 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:40:13.871 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:40:13.901 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:40:13.928 DEBUG:teuthology.nuke:shortname: smithi052 2024-03-25T19:40:13.928 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:40:13.951 DEBUG:teuthology.nuke:shortname: smithi179 2024-03-25T19:40:13.951 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:40:13.954 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:13.771034', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFu75LBK5+mmCXqXFQkEaeqx9olToVK74M8E1goFv+hIIYcWKlNBJAmGrZuaHX/fnZlzu8o4Ef/fcApmHnY2UWA='} 2024-03-25T19:40:14.001 INFO:teuthology.orchestra.console.smithi052:Power off 2024-03-25T19:40:14.001 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:40:14.012 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:13.772582', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPoGCLeTMdvJXt4cXBSaRu+M6PrJ8QIjxvJD4/CWoPQiXProONOev5+Tv0UkCP3AYZH/yTtc4C7WCDgX2lCfAyY='} 2024-03-25T19:40:14.038 DEBUG:teuthology.orchestra.console.smithi052:power off output: Chassis Power Control: Down/Off 2024-03-25T19:40:14.038 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:40:14.064 INFO:teuthology.orchestra.console.smithi179:Power off 2024-03-25T19:40:14.065 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:40:14.074 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-03-25T19:40:14.086 DEBUG:teuthology.orchestra.console.smithi179:power off output: Chassis Power Control: Down/Off 2024-03-25T19:40:14.086 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:40:14.108 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-25T19:40:18.076 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:40:18.109 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:40:18.185 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is on 2024-03-25T19:40:18.219 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-25T19:40:22.189 DEBUG:teuthology.orchestra.console.smithi052:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:40:22.220 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:40:22.300 DEBUG:teuthology.orchestra.console.smithi052:check power output: Chassis Power is off 2024-03-25T19:40:22.331 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-25T19:40:22.401 INFO:teuthology.orchestra.console.smithi052:Power off completed 2024-03-25T19:40:26.334 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:40:26.445 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is off 2024-03-25T19:40:26.546 INFO:teuthology.orchestra.console.smithi179:Power off completed 2024-03-25T19:40:26.653 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6334.77091383934 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1462e99bbde64254ac7447b7d77a04bd status: fail success: false 2024-03-25T19:40:26.653 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:40:26.745 INFO:teuthology.run:FAIL